Enable AF_UNIX support in Windows

[54b8045a-ddbb-4ba8-a97f-d3f1db53e980]

BPO	33408
Nosy	@pfmoore, @ericvsmith, @giampaolo, @tjguk, @zware, @zooba, @graingert, @animalize, @filips123, @paulmon, @websurfer5
PRs	python/cpython#14823

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-05-02.10:35:32.035>
labels = ['type-feature', '3.9', 'OS-windows']
title = 'Enable AF_UNIX support in Windows'
updated_at = <Date 2021-06-26.22:43:15.267>
user = 'https://github.com/filips123'

bugs.python.org fields:

activity = <Date 2021-06-26.22:43:15.267>
actor = 'graingert'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2018-05-02.10:35:32.035>
creator = 'filips123'
dependencies = []
files = []
hgrepos = []
issue_num = 33408
keywords = ['patch']
message_count = 9.0
messages = ['316067', '347629', '347645', '347646', '348080', '348880', '348913', '348923', '348936']
nosy_count = 11.0
nosy_names = ['paul.moore', 'eric.smith', 'giampaolo.rodola', 'tim.golden', 'zach.ware', 'steve.dower', 'graingert', 'malin', 'filips123', 'Paul Monson', 'Jeffrey.Kintscher']
pr_nums = ['14823']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue33408'
versions = ['Python 3.9']

[54b8045a-ddbb-4ba8-a97f-d3f1db53e980]

Unix socket (AF_UNIX) is now avalible in Windows 10 (April 2018 Update). Please add Python support for it. More details about it on https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/

[eef812a9-0c55-49e1-85e3-717a8588c91c]

I've been asked by my team to investigate what is required to enable AF_UNIX in Python.

Is anyone else actively investigating this? I did a prelinary investigation and the impact on test in test_sockets was pretty simple.

However there were 26 test failures in test_asyncio tests, with only the naive changes to all AF_UNIX to work with the socket class on Windows.

The first failure I looked at was caused by ProactorEventLoop in windows_events.py not providing a windows-specific version of create_unix_server. This results in the code falling back to AbstractServer.create_unix_server which raises NotImplementedError.

[b8e5a65c-bed4-4c5d-ba32-799f883ba638]

Have you upgraded the building SDK that supports AF_UNIX?

And should remove AF_UNIX flag at runtime on systems older than Windows 10 1804, see bpo-32394.

[b8e5a65c-bed4-4c5d-ba32-799f883ba638]

It would be nice to investigate the habit of using AF_UNIX in Python code on GitHub: https://github.com/search?l=Python&q=AF_UNIX&type=Code

If adding this flag will break a lot of code, due to lacking supports to datagram, maybe we should not add AF_UNIX at once, and waiting for full support to AF_UNIX.

[eef812a9-0c55-49e1-85e3-717a8588c91c]

I don't know if datagram support is coming to AF_UNIX on Windows.

The changes will only add the flag on Windows, and will enable stream sockets to use AF_UNIX on Windows. In mind this isn't breaking datagram support. It is true that it's a subset of what appears to be supported on Linux/Unix.

[b8e5a65c-bed4-4c5d-ba32-799f883ba638]

The current AF_UNIX address family of Windows10 doesn't support datagram, adding this flag may break some cross-platform code:

https://github.com/osbuild/osbuild/blob/9371eb9eaa3d0a7cab876eb4c7b70f519dfbd915/osbuild/init.py#L253 https://github.com/watsona4/dot_julia2/blob/5b7632b8089be01af706d8152555e711e0a81f06/conda/3/pkgs/python-3.7.3-h0371630_0/lib/python3.7/logging/handlers.py#L676 https://github.com/bognikol/Eleusis/blob/828672492b9cdb3444ddf466c0ac3055572277f7/Dependencies/02_macOS/40_gtk%2B/x64/lib/python2.7/test/test_socketserver.py#L177 https://github.com/sarnautov/python3/blob/005d75d8ac2b11a06707ce0184021a727d6fe844/lib/python3.6/test/test_asyncio/test_unix_events.py#L311 https://github.com/yanlianglai/ssr_python_leeblog/blob/8113fe3f32a250cc52e0cddad9761692dd840967/shadowsocks/manager.py#L60 ...

So I'm -1 on adding AF_UNIX personally.

I thought a compromise proposal:

• if (building SDK >= 1804 and run-time Windows >= 1804), add a undocumented flag _WIN_AF_UNIX, its value is equal to AF_UNIX.
• only use _WIN_AF_UNIX in stdlib, this is a black-box optimization.
• if someone really need high performance, he/she can uses this undocumented flag privately.

If one day Windows support full AF_UNIX, we can remove _WIN_AF_UNIX and add standard AF_UNIX.

[zooba]

Most of those examples would break today if run on Windows, though (AttributeError). So they'd just continue to break, probably with a different error (I'm not clear what happens if we specify SOCK_DGRAM with this change).

Having an undocumented field doesn't really help much - what we want is a documented field with a different name, perhaps WIN_AF_UNIX? (I think we need to keep the "AF_" prefix for real values and not mess up the namespace ourselves.) That way we can document that it matches AF_UNIX when defined on Windows, but will not trigger existing code that checks for the presence of AF_UNIX. Then we'll have to update the standard library to use either/both values where supported.

[eef812a9-0c55-49e1-85e3-717a8588c91c]

If you try to create a datagram socket with the current AF_UNIX changes on Windows the error is:

OSError: [WinError 10047] An address incompatible with the requested protocol was used

All of the examples given will fail to load with AttributeError on Windows because socket.AF_UNIX is not a valid attribute on Windows.

I think it's possible that there is code that detects whether AF_UNIX is an attribute on socket or not and does something different on Windows, but I haven't seen an example of it so far.

I can make the changes to WIN_AF_UNIX when I continue working on this, just in case there is portable code out there somewhere.

[zooba]

I think it's possible that there is code that detects whether AF_UNIX is an attribute on socket or not and does something different on Windows, but I haven't seen an example of it so far.

One of the test suites linked above had a skipUnless(hasattr(socket, 'AF_UNIX')) on the whole class, and at least one test assumed that it could use datagram support, so there's the example.

[futscdav]

This was really surprising to me. I have a native app that exposes a unix domain socket on windows for IPC, and wanted to write a Python client for it, only to find that the default socket interface does not allow using the AF_UNIX family on windows. Is there a 3rd party library that can work around this?

[robertlagrant]

@paulmon did you have any luck with this?

[paulmon]

This work wasn't completed because datagram support for AF_UNIX sockets on windows didn't exist when I made the original pull request.

I did a quick search and I don't think that has changed.

See this blog for more info about AF_UNIX support (not sure if the info is current or not) : https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/

[futscdav]

From what I understand, datagram support for AF_UNIX on windows is not planned. SOCK_STREAM is the only supported mode, but that's obviously already very useful in practice.

[leiless]

Any update on this?

[xavierog]

This issue stems from that blog post: https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/ That blog post's author stated that, contrary to what he had initially announced, abstract Unix sockets are not and will not be supported in Windows: https://github.com/microsoft/WSL/issues/4240#issuecomment-620805115

Overall, 2260 days after that blog post, as far as I know:

• AF_UNIX+SOCK_STREAM is less capable that announced/expected;
• there is no AF_UNIX+SOCK_DGRAM implementation in sight;
• SCM_RIGHTS was mentioned but not implemented;
• SOCK_SEQPACKET was not even mentioned;
• same for SCM_CREDENTIALS (but that mechanism is usually very OS-specific and Python does not exactly support it).

At this stage, and unless Microsoft publishes a new shiny blog post expressing their firm intent to further develop AF_UNIX sockets in their operating system, it seems fair to "time out" and proceed with an ad-hoc implementation.

I do not have any strong opinion regarding AF_UNIX vs WIN_AF_UNIX. I would suggest to add AF_UNIX_PARTIALSUPPORT and document that using this family may raise a specific exception (e.g. a class akin to NotImplementedError?). This way, AF_UNIX keeps its current meaning of "this system offers decent support for multiple flavours of AF_UNIX sockets" and AF_UNIX_PARTIALSUPPORT users know what to expect.

[ErikHons]

AF_UNIX+SOCK_STREAM would let dbus-next talk to the freedesktop reference server over a UDS like other clients. I believe all of the identification options would work fine there.

[lestie198]

@futscdav

This was really surprising to me. I have a native app that exposes a unix domain socket on windows for IPC, and wanted to write a Python client for it, only to find that the default socket interface does not allow using the AF_UNIX family on windows. Is there a 3rd party library that can work around this?

I have a similar issue, did you find a solution?

[xavierog]

Is there a 3rd party library that can work around this?

I have a similar issue, did you find a solution?

Theoretically, it should be possible to leverage ctypes[1] to reach the WinSock API and create AF_UNIX sockets. But I doubt the resulting socket could ever be shoehorned back into the socket module, meaning all other operations (send, receive, close, ...) would need to be done through ctypes+WinSock too. This is presumably a steep price compared to most practical applications.

[1] -- correct me if I am wrong but it seems the PyWin32 project does not expose the WinSock API.

[RivenSkaye]

Another year has passed and this seems to still be dead in the water, including the part where they were going to change the linked article to not list abstract socket support. Unnamed sockets are supported, except for the minor detail that socketpair(2) isn't. It is unclear, however, if this could be hacked together using socket(2) calls.

Effectively AF_UNIX is limited to pathname-addressed sockets. Regardless of naming, I am in favor of a clearly different name to indicate only a minimal API. Bonus points if it's portable across platforms.
Pros of AF_UNIX_PARTIAL:

• As a strict subset of AF_UNIX it could be used to write platform independent code (for recent enough Windows builds)
• Very clearly conveys the fact it's limited
• If it were made an ABC or similar, and MS development ever continues, it won't clash with existing AF_UNIX bits and pieces in other implementations
  • This would also allow users to roll their own through ctypes without breaking API compat

Cons:

• Doesn't clearly identify itself as Windows specific (though this might be a pro if it makes it usable for other OSes?)
• Might need a lot of plumbing to appear the same across platforms