aio-libs
AttributeError: 'NoneType' object has no attribute 'get_extra_info' with aiohttp==3.8.1 when using a proxy
Describe the bug
Requests involving a proxy server sometimes fail with an AttributeError which were working on versions prior to 3.8.0
To Reproduce
Full test case:
# test_case.py
import asyncio
import aiohttp
async def test_case():
async with aiohttp.ClientSession() as session:
async with session.get("https://api.ipify.org", proxy="http://218.235.10.214:8393") as response:
text = await response.text()
print(text)
def main():
asyncio.run(test_case())
if __name__ == "__main__":
main()
Expected behavior
Expected behavior (confirmed working with aiohttp==3.7.4.post0)
python3 test_case.py
<prints my public ip>
Logs/tracebacks
Traceback (most recent call last):
File "/home/john/Code/projects/proxywrench/proxywrench/test_case.py", line 17, in <module>
main()
File "/home/john/Code/projects/proxywrench/proxywrench/test_case.py", line 13, in main
asyncio.run(test_case())
File "/home/john/.pyenv/versions/3.9.7/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/home/john/.pyenv/versions/3.9.7/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/home/john/Code/projects/proxywrench/proxywrench/test_case.py", line 6, in test_case
async with session.get("https://api.ipify.org", proxy="http://218.235.10.214:8393") as response:
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 1140, in __aenter__
self._resp = await self._coro
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 543, in connect
proto = await self._create_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 904, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 1324, in _create_proxy_connection
return await self._start_tls_connection(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 1125, in _start_tls_connection
tls_proto.connection_made(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/base_protocol.py", line 58, in connection_made
tcp_nodelay(tr, True)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/tcp_helpers.py", line 26, in tcp_nodelay
sock = transport.get_extra_info("socket")
AttributeError: 'NoneType' object has no attribute 'get_extra_info'
Python Version
$ python --version
Python 3.9.7
aiohttp Version
$ python -m pip show aiohttp
Name: aiohttp
Version: 3.8.0
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author:
Author-email:
License: Apache 2
Location: /home/john/.pyenv/versions/3.9.7/envs/proxywrench/lib/python3.9/site-packages
Requires: aiosignal, async-timeout, attrs, charset-normalizer, frozenlist, multidict, yarl
Required-by: aiohttp-socks, aiohttp-utils
multidict Version
$ python -m pip show multidict
Name: multidict
Version: 5.1.0
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /home/john/.pyenv/versions/3.9.7/envs/proxywrench/lib/python3.9/site-packages
Requires:
Required-by: aiohttp, yarl
yarl Version
$ python -m pip show yarl
Name: yarl
Version: 1.6.3
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /home/john/.pyenv/versions/3.9.7/envs/proxywrench/lib/python3.9/site-packages
Requires: idna, multidict
Required-by: aiohttp
OS
Ubuntu 21.10 x64
Related component
Client
Additional context
No response
Code of Conduct
- [X] I agree to follow the aio-libs Code of Conduct
@bmbouter @jborean93 have you seen this problem after the proxy code update?
I kind of think I did, but I tested so many permutations it's difficult to say 100%. If I am remembering correctly, if TLS trust was required but one or both of the certs was untrusted from the local trust store, I would get this traceback. Hopefully this is helpful.
@bmbouter could you check if the reproducer above works for you locally?
On my machine it's
Traceback (most recent call last):
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/connector.py", line 985, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore[return-value] # noqa
File "~/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 1056, in create_connection
raise exceptions[0]
File "~/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 1041, in create_connection
sock = await self._connect_sock(
File "~/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 955, in _connect_sock
await self.sock_connect(sock, address)
File "~/.pyenv/versions/3.9.0/lib/python3.9/asyncio/selector_events.py", line 502, in sock_connect
return await fut
File "~/.pyenv/versions/3.9.0/lib/python3.9/asyncio/selector_events.py", line 537, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
TimeoutError: [Errno 110] Connect call failed ('218.235.10.214', 8393)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "~/.pyenv/versions/3.9.0/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "~/.pyenv/versions/3.9.0/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "~/src/experiments/aiohttp-repros/issue-6239/test_case.py", line 17, in <module>
main()
File "~/src/experiments/aiohttp-repros/issue-6239/test_case.py", line 13, in main
asyncio.run(test_case())
File "~/.pyenv/versions/3.9.0/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "~/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "~/src/experiments/aiohttp-repros/issue-6239/test_case.py", line 6, in test_case
async with session.get("https://api.ipify.org", proxy="http://218.235.10.214:8393") as response:
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/client.py", line 1140, in __aenter__
self._resp = await self._coro
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/connector.py", line 543, in connect
proto = await self._create_connection(req, traces, timeout)
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/connector.py", line 904, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/connector.py", line 1230, in _create_proxy_connection
transport, proto = await self._create_direct_connection(
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/connector.py", line 1205, in _create_direct_connection
raise last_exc
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/connector.py", line 1174, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/connector.py", line 991, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientProxyConnectionError: Cannot connect to host 218.235.10.214:8393 ssl:default
(looks like the initial report may have missed the first part of the traceback)
same error here.
File "/Users/dev/Library/Caches/pypoetry/virtualenvs/proj/lib/python3.7/site-packages/aiohttp/client.py", line 536, in _request
req, traces=traces, timeout=real_timeout
File "/Users/dev/Library/Caches/pypoetry/virtualenvs/proj/lib/python3.7/site-packages/aiohttp/connector.py", line 543, in connect
proto = await self._create_connection(req, traces, timeout)
File "/Users/dev/Library/Caches/pypoetry/virtualenvs/proj/lib/python3.7/site-packages/aiohttp/connector.py", line 904, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
File "/Users/dev/Library/Caches/pypoetry/virtualenvs/proj/lib/python3.7/site-packages/aiohttp/connector.py", line 1329, in _create_proxy_connection
timeout=timeout,
File "/Users/dev/Library/Caches/pypoetry/virtualenvs/proj/lib/python3.7/site-packages/aiohttp/connector.py", line 1126, in _start_tls_connection
tls_transport
File "/Users/dev/Library/Caches/pypoetry/virtualenvs/proj/lib/python3.7/site-packages/aiohttp/base_protocol.py", line 58, in connection_made
tcp_nodelay(tr, True)
File "/Users/dev/Library/Caches/pypoetry/virtualenvs/proj/lib/python3.7/site-packages/aiohttp/tcp_helpers.py", line 26, in tcp_nodelay
sock = transport.get_extra_info("socket")
AttributeError: 'NoneType' object has no attribute 'get_extra_info'
@bmbouter could you check if the reproducer above works for you locally? (looks like the initial report may have missed the first part of the traceback)
I don't think I did. That proxy server is no longer online. That's why you're getting a Timeout now.
Let me grab another one that reproduces the original issue.
Here's the revised test case with a different proxy.
# test_case.py
import asyncio
import aiohttp
async def test_case():
async with aiohttp.ClientSession() as session:
async with session.get("https://api.ipify.org", proxy="http://42.194.222.36:3128") as response:
text = await response.text()
print(text)
def main():
asyncio.run(test_case())
if __name__ == "__main__":
main()
Here's the copy/pasted output from my terminal. (Full, not truncated.)
(proxywrench) john@john-work:~/Code/projects/proxywrench$ python test_case.py
Traceback (most recent call last):
File "/home/john/Code/projects/proxywrench/test_case.py", line 18, in <module>
main()
File "/home/john/Code/projects/proxywrench/test_case.py", line 14, in main
asyncio.run(test_case())
File "/home/john/.pyenv/versions/3.9.7/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/home/john/.pyenv/versions/3.9.7/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/home/john/Code/projects/proxywrench/test_case.py", line 7, in test_case
async with session.get("https://api.ipify.org", proxy="http://42.194.222.36:3128") as response:
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 1140, in __aenter__
self._resp = await self._coro
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 543, in connect
proto = await self._create_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 904, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 1324, in _create_proxy_connection
return await self._start_tls_connection(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 1125, in _start_tls_connection
tls_proto.connection_made(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/base_protocol.py", line 58, in connection_made
tcp_nodelay(tr, True)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/tcp_helpers.py", line 26, in tcp_nodelay
sock = transport.get_extra_info("socket")
AttributeError: 'NoneType' object has no attribute 'get_extra_info'
Here's a screenshot of my terminal:
Here's two additional proxies that exhibit the same behavior:
http://42.194.222.36:3128
http://178.32.107.236:3128
http://101.34.116.37:7890
Note: These proxies may or may not be broken. I'm not totally sure. The real bug isn't that they were necessarily working before 3.8.0, but rather that before 3.8.0 it gave a much more helpful exception than the generic AttributeError.
I don't want my code to have to catch AttributeError because that could mask all kinds of other issues.
Here's a much more detailed output.
Here's the script I'm running
# test_case.py
import asyncio
import traceback
import aiohttp
PROXIES = [
"http://42.194.222.36:3128",
"http://178.32.107.236:3128",
"http://101.34.116.37:7890",
]
async def test_all():
async with aiohttp.ClientSession() as session:
for proxy in PROXIES:
try:
async with session.get("https://api.apify.org", proxy=proxy) as response:
text = await response.text()
except Exception as e:
print(f"{proxy} : Error")
print(traceback.format_exc())
else:
print(f"{proxy} : OK")
def main():
asyncio.run(test_all())
if __name__ == "__main__":
main()
Here's the output on aiohttp==3.7.4.post0
(Note the very helpful ConnectionResetError, ClientConnectionError, SSLError, ClientConnectorSSLError, etc.)
(proxywrench) john@john-work:~/Code/projects/proxywrench$ python test_case.py
http://42.194.222.36:3128 : Error
Traceback (most recent call last):
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 969, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
File "/home/john/.pyenv/versions/3.9.7/lib/python3.9/asyncio/base_events.py", line 1081, in create_connection
transport, protocol = await self._create_connection_transport(
File "/home/john/.pyenv/versions/3.9.7/lib/python3.9/asyncio/base_events.py", line 1111, in _create_connection_transport
await waiter
ConnectionResetError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/john/Code/projects/proxywrench/test_case.py", line 18, in test_all
async with session.get("https://api.apify.org", proxy=proxy) as response:
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 1117, in __aenter__
self._resp = await self._coro
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 520, in _request
conn = await self._connector.connect(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 535, in connect
proto = await self._create_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 890, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 1139, in _create_proxy_connection
transport, proto = await self._wrap_create_connection(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 975, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.apify.org:443 ssl:default [None]
http://178.32.107.236:3128 : Error
Traceback (most recent call last):
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 969, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
File "/home/john/.pyenv/versions/3.9.7/lib/python3.9/asyncio/base_events.py", line 1081, in create_connection
transport, protocol = await self._create_connection_transport(
File "/home/john/.pyenv/versions/3.9.7/lib/python3.9/asyncio/base_events.py", line 1111, in _create_connection_transport
await waiter
File "/home/john/.pyenv/versions/3.9.7/lib/python3.9/asyncio/sslproto.py", line 528, in data_received
ssldata, appdata = self._sslpipe.feed_ssldata(data)
File "/home/john/.pyenv/versions/3.9.7/lib/python3.9/asyncio/sslproto.py", line 188, in feed_ssldata
self._sslobj.do_handshake()
File "/home/john/.pyenv/versions/3.9.7/lib/python3.9/ssl.py", line 944, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1129)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/john/Code/projects/proxywrench/test_case.py", line 18, in test_all
async with session.get("https://api.apify.org", proxy=proxy) as response:
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 1117, in __aenter__
self._resp = await self._coro
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 520, in _request
conn = await self._connector.connect(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 535, in connect
proto = await self._create_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 890, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 1139, in _create_proxy_connection
transport, proto = await self._wrap_create_connection(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 973, in _wrap_create_connection
raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host api.apify.org:443 ssl:default [[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1129)]
http://101.34.116.37:7890 : Error
Traceback (most recent call last):
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 969, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
File "/home/john/.pyenv/versions/3.9.7/lib/python3.9/asyncio/base_events.py", line 1081, in create_connection
transport, protocol = await self._create_connection_transport(
File "/home/john/.pyenv/versions/3.9.7/lib/python3.9/asyncio/base_events.py", line 1111, in _create_connection_transport
await waiter
ConnectionResetError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/john/Code/projects/proxywrench/test_case.py", line 18, in test_all
async with session.get("https://api.apify.org", proxy=proxy) as response:
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 1117, in __aenter__
self._resp = await self._coro
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 520, in _request
conn = await self._connector.connect(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 535, in connect
proto = await self._create_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 890, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 1139, in _create_proxy_connection
transport, proto = await self._wrap_create_connection(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 975, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.apify.org:443 ssl:default [None]
Here's the output with aiohttp==3.8.0
Note that all of the helpful exceptions disappeared.
(proxywrench) john@john-work:~/Code/projects/proxywrench$ python test_case.py
http://42.194.222.36:3128 : Error
Traceback (most recent call last):
File "/home/john/Code/projects/proxywrench/test_case.py", line 18, in test_all
async with session.get("https://api.apify.org", proxy=proxy) as response:
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 1140, in __aenter__
self._resp = await self._coro
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 543, in connect
proto = await self._create_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 904, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 1324, in _create_proxy_connection
return await self._start_tls_connection(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 1125, in _start_tls_connection
tls_proto.connection_made(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/base_protocol.py", line 58, in connection_made
tcp_nodelay(tr, True)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/tcp_helpers.py", line 26, in tcp_nodelay
sock = transport.get_extra_info("socket")
AttributeError: 'NoneType' object has no attribute 'get_extra_info'
http://178.32.107.236:3128 : Error
Traceback (most recent call last):
File "/home/john/Code/projects/proxywrench/test_case.py", line 18, in test_all
async with session.get("https://api.apify.org", proxy=proxy) as response:
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 1140, in __aenter__
self._resp = await self._coro
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 543, in connect
proto = await self._create_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 904, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 1324, in _create_proxy_connection
return await self._start_tls_connection(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 1125, in _start_tls_connection
tls_proto.connection_made(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/base_protocol.py", line 58, in connection_made
tcp_nodelay(tr, True)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/tcp_helpers.py", line 26, in tcp_nodelay
sock = transport.get_extra_info("socket")
AttributeError: 'NoneType' object has no attribute 'get_extra_info'
http://101.34.116.37:7890 : Error
Traceback (most recent call last):
File "/home/john/Code/projects/proxywrench/test_case.py", line 18, in test_all
async with session.get("https://api.apify.org", proxy=proxy) as response:
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 1140, in __aenter__
self._resp = await self._coro
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 543, in connect
proto = await self._create_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 904, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 1324, in _create_proxy_connection
return await self._start_tls_connection(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/connector.py", line 1125, in _start_tls_connection
tls_proto.connection_made(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/base_protocol.py", line 58, in connection_made
tcp_nodelay(tr, True)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.9/site-packages/aiohttp/tcp_helpers.py", line 26, in tcp_nodelay
sock = transport.get_extra_info("socket")
AttributeError: 'NoneType' object has no attribute 'get_extra_info'
I don't want my code to have to catch
AttributeErrorbecause that could mask all kinds of other issues.
Fair enough. FWIW the best place to add regression tests is https://github.com/aio-libs/aiohttp/blob/master/tests/test_proxy_functional.py.
Would you accept a pull request with a failing test, but no fix?
I don't know how to fix this issue, but I could definitely add the test.
Would you accept a pull request with a failing test, but no fix?
Yes, we do this frequently by marking it with xfail, so we can come back to fix it later on.
Would you accept a pull request with a failing test, but no fix?
I don't know how to fix this issue, but I could definitely add the test.
Yes, we're trying to follow https://pganssle-talks.github.io/xfail-lightning/.
this commit breaks - https://github.com/aio-libs/aiohttp/commit/cf641aa01aed6728717bd4abc306aaeb6d719573
After replace https://github.com/aio-libs/aiohttp/blob/3.8/aiohttp/connector.py#L1212
runtime_has_start_tls = self._loop_supports_start_tls() -> runtime_has_start_tls = False
proxy works fine
So it looks like those proxies are trying to support HTTPS (TLS-in-TLS?) but there's some issue with the configuration?
If that's the case, a specific exception(s) would be ideal there
perhaps something similar needs to be done here
runtime_has_start_tls = False if req.proxy.scheme != "https" else self._loop_supports_start_tls()
No, I'm not familiar with this code. If someone wants to create a PR with a fix, we can take a look at it.
Also doesn't look like anybody created a PR with a test yet, which was suggested before (which makes it more likely that one of the maintainers will then have time to find a fix).
So it looks like those proxies are trying to support HTTPS (TLS-in-TLS?) but there's some issue with the configuration?
Sort of. TLS-in-TLS is tricky in CPython itself and requires monkey-patching of the stdlib to make it work, this is why it's considered a "tech preview". Before the patch, loop.start_tls() wasn't used at all but after this change, it should be used in monkey-patched envs for TLS-in-TLS and maybe for HTTPS over plain HTTP proxy IIRC.
FWIW aiohttp could benefit from more regression tests. Plus a lot of those that exist could be more useful if rewritten using the new proxy.py-based pytest fixture instead of relying on mocks and monkey-patching.
If that's the case, a specific exception(s) would be ideal there
I'm open to proposals.
If that's the case, a specific exception(s) would be ideal there
I'm open to proposals.
If it could raise aiohttp.client_exceptions.ClientConnectorError, that would "fix" this issue as far as I'm concerned.
It's still an issue in 3.8.1
I am experience this issue as well, until it is fixed what is the best way to catch this error?
As of late I have been experiencing this issue as well using private proxies from the provider "Oxylabs", anyone come up with a fix yet? the edits in the above pr help with providing a better error message but I'm curious as to what is different with these proxies compared to the many others I've used without issue (these actually worked fine just a week ago but now seemingly only dont work in aiohttp)
This https://github.com/aio-libs/aiohttp/pull/6703 does appear to fix it, but I had trouble getting master to build with that patch. Someone smarter than me should look into it more.
I still see same issue 'NoneType' object has no attribute 'get_extra_info' . I used requests.get with this proxy normally.
Here's an update as of today. Using python3.11 on ubuntu. New tracebacks make this much easier to read.
Test Code
import asyncio
import aiohttp
async def test_case():
async with aiohttp.ClientSession() as session:
async with session.get(
"https://api.ipify.org", proxy="http://185.38.111.1:8080"
) as response:
text = await response.text()
print(text)
def main():
asyncio.run(test_case())
if __name__ == "__main__":
main()
Expected / Correct Behavior on aiohttp==3.7.4.post0
The code raises an exception that I can catch to recover.
Traceback (most recent call last):
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/connector.py", line 969, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/3.11.0/lib/python3.11/asyncio/base_events.py", line 1098, in create_connection
transport, protocol = await self._create_connection_transport(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/3.11.0/lib/python3.11/asyncio/base_events.py", line 1131, in _create_connection_transport
await waiter
File "/home/john/.pyenv/versions/3.11.0/lib/python3.11/asyncio/sslproto.py", line 577, in _on_handshake_complete
raise handshake_exc
File "/home/john/.pyenv/versions/3.11.0/lib/python3.11/asyncio/sslproto.py", line 559, in _do_handshake
self._sslobj.do_handshake()
File "/home/john/.pyenv/versions/3.11.0/lib/python3.11/ssl.py", line 979, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:992)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/john/Code/proxywrench/proxywrench/test_case.py", line 20, in <module>
main()
File "/home/john/Code/proxywrench/proxywrench/test_case.py", line 16, in main
asyncio.run(test_case())
File "/home/john/.pyenv/versions/3.11.0/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/3.11.0/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/3.11.0/lib/python3.11/asyncio/base_events.py", line 650, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/home/john/Code/proxywrench/proxywrench/test_case.py", line 7, in test_case
async with session.get(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/client.py", line 1117, in __aenter__
self._resp = await self._coro
^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/client.py", line 520, in _request
conn = await self._connector.connect(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/connector.py", line 535, in connect
proto = await self._create_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/connector.py", line 890, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/connector.py", line 1139, in _create_proxy_connection
transport, proto = await self._wrap_create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/connector.py", line 973, in _wrap_create_connection
raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host api.ipify.org:443 ssl:default [[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:992)]
Unexpected / Incorrect behavior with aiohttp-3.8.3
A generic AttributeError is raised. If I catch this AttributeError, I will potentially handle unrelated errors.
Traceback (most recent call last):
File "/home/john/Code/proxywrench/proxywrench/test_case.py", line 20, in <module>
main()
File "/home/john/Code/proxywrench/proxywrench/test_case.py", line 16, in main
asyncio.run(test_case())
File "/home/john/.pyenv/versions/3.11.0/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/3.11.0/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/3.11.0/lib/python3.11/asyncio/base_events.py", line 650, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/home/john/Code/proxywrench/proxywrench/test_case.py", line 7, in test_case
async with session.get(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in __aenter__
self._resp = await self._coro
^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/client.py", line 536, in _request
conn = await self._connector.connect(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/connector.py", line 540, in connect
proto = await self._create_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/connector.py", line 899, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/connector.py", line 1325, in _create_proxy_connection
return await self._start_tls_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/connector.py", line 1124, in _start_tls_connection
tls_proto.connection_made(
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/base_protocol.py", line 58, in connection_made
tcp_nodelay(tr, True)
File "/home/john/.pyenv/versions/proxywrench/lib/python3.11/site-packages/aiohttp/tcp_helpers.py", line 25, in tcp_nodelay
sock = transport.get_extra_info("socket")
^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get_extra_info'
Edit
The type annotation here https://github.com/aio-libs/aiohttp/blob/7d78fd01dbe983d119141d7f2775aefd42494f99/aiohttp/tcp_helpers.py#L24 suggests that tcp_no_delay should never be called with None as the first parameter. So the base_protocol function is violating the type signature of that function.
It looks like transport here is None https://github.com/aio-libs/aiohttp/blob/7d78fd01dbe983d119141d7f2775aefd42494f99/aiohttp/base_protocol.py#L56
Appears this code is returning None https://github.com/aio-libs/aiohttp/blob/d7ebbeb9883c8f1165176e9dcf5161b073306233/aiohttp/connector.py#L1036-L1042
Edit 2
Here's the start_tls code in cpython
https://github.com/python/cpython/blob/8079bef56f2249ecedafe0be5a6d7a120a7f3ac3/Lib/asyncio/base_events.py#L1231-L1277
Looks like it will return None if ssl_protocol._app_transport is None and doesn't raise an exception
_app_transport is initialized in __init__ by this method: https://github.com/python/cpython/blob/8079bef56f2249ecedafe0be5a6d7a120a7f3ac3/Lib/asyncio/sslproto.py#L373-L379
But is explicitly set back to None if the connection is lost: https://github.com/python/cpython/blob/8079bef56f2249ecedafe0be5a6d7a120a7f3ac3/Lib/asyncio/sslproto.py#L414
perhaps something similar needs to be done here
runtime_has_start_tls = False if req.proxy.scheme != "https" else self._loop_supports_start_tls()
That help me, proxy is worked, thanks!
Same here, full stacktrace in https://github.com/aio-libs/aiohttp/issues/3355#issuecomment-1596099408. Is there any chance to get this fixed? Regression is two years old now.
I don't know if this could help fixing the problem but it works on Windows. It fails on debian for me
