MagicStack
Transport sockets do not support shutdown() method
- uvloop version: 0.12.2
- Python version: 3.5.3
- Platform: Debian 9
-
Can you reproduce the bug with
PYTHONASYNCIODEBUGin env?: yes
An attempt to use shutdown method of transport socket causes following exception:
source_traceback: Object created at (most recent call last):
File "/usr/local/bin/ssh-tarpit", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.5/dist-packages/ssh_tarpit/__main__.py", line 109, in main
loop.run_until_complete(amain(args, loop))
File "/usr/lib/python3.5/asyncio/streams.py", line 238, in connection_made
self._stream_writer)
File "/usr/local/lib/python3.5/dist-packages/ssh_tarpit/server.py", line 52, in _spawn
self._loop.create_task(self.handler(reader, writer)))
Traceback (most recent call last):
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/usr/lib/python3.5/asyncio/coroutines.py", line 125, in send
return self.gen.send(value)
File "/usr/local/lib/python3.5/dist-packages/ssh_tarpit/server.py", line 38, in handler
sock.shutdown(socket.SHUT_RD)
File "uvloop/pseudosock.pyx", line 155, in uvloop.loop.PseudoSocket.shutdown
File "uvloop/pseudosock.pyx", line 19, in uvloop.loop.PseudoSocket._na
TypeError: transport sockets do not support shutdown() method
Such behavior is not observed with default event loop.
Such behavior is not observed with default event loop.
It will be in 3.8 or later. You're using an undocumented way of shutdowning the socket bypassing the top-level transport API.
@1st1 Hello, thanks for answer!
No, I'm interested in SHUT_RD, so I use sock.shutdown(socket.SHUT_RD) after transport.pause_reading().
Well, there's no API for SHUT_RD in libuv, so uvloop can't support it in any way :(
Why do you need SHUT_RD? It's not a coincidence it's not supported directly in asyncio/twisted; AFAIK it's pretty much useless, because if the the other side continues to send you data you will receive it anyways.
because if the the other side continues to send you data you will receive it anyways
TCP has flow control and sender shall not send more than TCP window size allows him. In this case TCP window will have zero size:
I just tested it with standard Python 3 event loop and everything works as expected. Also I used strace to confirm shutdown syscall was issued and target file descriptor was deregistered from epoll:
12087 getsockname(7, {sa_family=AF_INET, sin_port=htons(2222), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
12087 setsockopt(7, SOL_TCP, TCP_NODELAY, [1], 4) = 0
12087 epoll_wait(3, [], 2, 0) = 0
12087 epoll_ctl(3, EPOLL_CTL_ADD, 7, {EPOLLIN, {u32=7, u64=94158568030215}}) = 0
12087 epoll_wait(3, [], 3, 0) = 0
12087 epoll_ctl(3, EPOLL_CTL_DEL, 7, 0x7ffeca04d0d4) = 0
12087 shutdown(7, SHUT_RD) = 0
12087 getpid() = 12087
12087 write(2, "2019-03-23 22:48:58 INFO TarpitServer: Client ('127.0.0.1', 47216) connected\n", 81) = 81
I consider such behavior as reference behavior: no new events on incoming data will occur, sender transmission will be stalled, data in opposite direction still passing.
shutdown(SHUT_RD) is a valid way to tell system stop accepting new data and potentially free some of associated resources. I may agree: real behavior may vary from system, kernel version, platform and actual socket type, but generally it is integral part of sockets API.
OK, fair enough.
The reason why transport.get_extra_info('socket') does not return a real socket object is because manipulating the socket FD without the loop knowing about it can disrupt the operation of the event loop (even segfault it some cases). This has been implemented in uvloop, and I plan to do the same in asyncio in 3.8 (or 3.9). Therefore, the way you use it to set SHUT_RD can stop working at some point of time. So I'd suggest you to open an issue on bugs.python.org to add a new transport-level asyncio API (e.g. something like transport.shutdown_reading()) to set SHUT_RD for the underlying socket. I can then implement that API in uvloop.
Meanwhile you can try this (untested):
tr_sock = transport.get_extra_info('socket')
sock = socket.socket(tr_sock.family, tr_sock.type, tr_sock.proto, tr_sock.fileno())
try:
sock.shutdown(socket.SHUT_RD)
finally:
sock.detach()
For others reference: following code confirmed to work.
async def handler(self, reader, writer):
writer.transport.pause_reading()
sock = writer.transport.get_extra_info('socket')
if sock is not None:
try:
sock.shutdown(socket.SHUT_RD)
except TypeError:
direct_sock = socket.socket(sock.family, sock.type, sock.proto, sock.fileno())
try:
direct_sock.shutdown(socket.SHUT_RD)
finally:
direct_sock.detach()
Only disturbing thing is fact EPOLL_CTL_DEL is issued after shutdown:
6053 epoll_wait(3, [], 1024, 0) = 0
6053 epoll_wait(3, [{EPOLLIN, {u32=12, u64=12}}], 1024, 500) = 1
6053 accept4(12, NULL, NULL, SOCK_CLOEXEC|SOCK_NONBLOCK) = 13
6053 setsockopt(13, SOL_TCP, TCP_NODELAY, [1], 4) = 0
6053 accept4(12, NULL, NULL, SOCK_CLOEXEC|SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
6053 getsockname(13, {sa_family=AF_INET, sin_port=htons(2222), sin_addr=inet_addr("127.0.0.1")}, [128->16]) = 0
6053 getpeername(13, {sa_family=AF_INET, sin_port=htons(38822), sin_addr=inet_addr("127.0.0.1")}, [128->16]) = 0
6053 epoll_ctl(3, EPOLL_CTL_ADD, 13, {EPOLLIN, {u32=13, u64=13}}) = 0
6053 epoll_wait(3, [], 1024, 0) = 0
6053 getsockname(13, {sa_family=AF_INET, sin_port=htons(2222), sin_addr=inet_addr("127.0.0.1")}, [128->16]) = 0
6053 shutdown(13, SHUT_RD) = 0
6053 getpid() = 6053
6053 write(2, "2019-03-26 21:41:07 INFO TarpitServer: Client ('127.0.0.1', 38822) connected\n", 81) = 81
6053 getpid() = 6053
6053 epoll_wait(3, [{EPOLLIN, {u32=13, u64=13}}], 1024, 21) = 1
6053 epoll_ctl(3, EPOLL_CTL_DEL, 13, 0x7fff10bf3fa0) = 0
6053 epoll_wait(3, [], 1024, 21) = 0
6053 getpid() = 6053
Presumably transport.pause_reading() works not immediately in uvloop and action taken only when event loop gains execution again. It is not clear, if socket read attempts may occur in that short period (which, in its turn, will definitely fail due to socket new state).