API Roadmap

[milseman]

This is meant to be a continuously-updated list of API candidates for System.

Criteria for Inclusion

System's primary aim is to be the place developers go to access native platform interfaces, presented in as Swifty a way as feasible. This positions System as a successor to Darwin/GlibC/SwiftWin32 and a “bedrock” upon which to enable better systems-level programming in Swift.

System is not aiming to be a cross-platform abstraction layer; there will be platform differences reflected in System's API. Of course, if a concept can be expressed the same way across platforms without harm or significant compromise to the developer's native experience on that platform, that's great. But, System's API should feel natural and native for the target platform.

System aims to provide API in 3 areas:

1. As-Swifty-as-we-reasonably-can direct access to system calls and types

If a system interface exists, that’s strong justification for providing access to it in System as faithfully as we can.

2. Common types for library API at the systems level

System hosts common types for systems level programming, enabling libraries built on top of System to use the same, well-crafted types in their API.

For example, many API take locations in the file system, for which FilePath provides a common expression.

3. High-value utility functionality and abstractions, where appropriate

System provides common utilities that users of System would otherwise find themselves having to reinvent. These usually have some combination of being pervasive, having obvious desired behavior, and being difficult/onerous to implement correctly.

For example, ensuring that a file descriptor is closed under all (non-pathological) conditions can be complex in the presence of error-handling, so System provides FileDescriptor.closeAfter.

This area has a higher bar for contribution, requiring more justification and thought. What is "obvious desired behavior" and "correct" can differ amongst users of System.

Roadmap

Released

• 0.0.1
  • syscalls: read, write, lseek, pread, pwrite, open, close
  • types: FilePath, FileDescriptor, Errno, FilePermissions
  • helpers: closeAfter, writeAll
• 0.0.2
  • FilePath Syntactic Operations
    • decompose a path, analyze a path, components view, mutate a path, etc
  • Standard file descriptors
    • static FileDescriptor.standardInput, FileDescriptor.standardOutput, FileDescriptor.standardError
  • dup/dup2
    • FileDescriptor.duplicate, with optional as target.

Merged

In Progress

• Strongly-typed process and signal wrappers
  • ProcessID, Signal, SignalSet, ProcessID.TaskInfo, ProcessID.ResourceUsageInfo
  • TODO: posix_spawn and execv functionality and interfaces
• Initial socket support
• Pipes

Sketches

Sketches are merge-worthy, quick-but-complete presentations of systems interfaces. They help to surface unknown-unknowns and can be easily adapted into a proposal. They are not necessarily named or expressed in their final form.

• fcntl
• sysconf, pathconf, fpathconf
• openat, fsync, sync

Starter tasks

• Add support for pipe

Near Term

Anything here could be added to System in the near term. Most of the functionality can be added by following existing API design patterns, though some will need new patterns.

• sleep capabilities (e.g. usleep)
• FilePath semantic (i.e. file-system interacting) operations
  • stat, chown, directory iteration, etc.
  • temporary files
• pthread low-level interfaces
• Further networking and sockets functionality
  • getaddrinfo, gethostent, etc.
• Environment variables
  • FilePath: ~ expansion, currentWorkingDirectory
• OSString-like abstraction
  • Null-terminated bytes on Unix, wchar or bytes on Windows (pending investigation)
• I/O events
  • kqueue/kevent for Darwin, epoll for Linux, APC (or something similar) for Windows
• exit handling
  • exit, atexit, etc
• tty
  • ioctl, etc
• Fleshing out FileDescriptor more
  • chmod, umask, etc.

Long Term (vague "blue-sky" hopes)

• moveonly File type
• high level Process type
• io_uring on Linux

[allenhumphreys]

Not sure if it's on the road map, but additional FileDescriptor operations such as dup and dup2 also support for the FILE APIs? I'm not 100% sure if this aligns with this packages goals, just commenting because I was recently doing some things with these.

[milseman]

@allenhumphreys you're right, I'll add those. Sorry that I didn't see your reply earlier, I sometimes miss Github notifications (still trying to figure out why).

[allenhumphreys]

Thanks @miles. For reference, here's where I used the package. It's a great example of where the API's coverage is quite low.

Really glad to see this package coming along though!

[milseman]

Updated to reflect merged API that is yet-to-be-released, and added in references to Standard file descriptors, and Strongly-typed process and signal wrappers.

[milseman]

Updated for dup/dup2

[GeorgeLyon]

Just curious, does pipe fit somewhere on this roadmap?

[milseman]

I had mentally lumped that under process types and calls, but it's not actually blocked on any design work there. We can add this anytime and might make a good starter task.

extension FileDescriptor {
  static func openPipe() throws -> (read: FileDescriptor, write: FileDescriptor)
}

[compnerd]

Pipes on Windows are somewhat interesting in that they behave differently. I think that it may be a good idea to ensure that we have a proper design to accommodate that properly.

[milseman]

@compnerd, would CreatePipe be an appropriate equivalent? It seems sensible to have an additional overload on Windows that takes a buffer size and perhaps security policy (or at least a Bool for inheritability). Buffering has a default value and we can pick a default regarding inheritance for the parameter-less (or defaulted) API.

[compnerd]

CreatePipe can be used to create the anonymous pipe and that is better because you can control inheritability unlike with _pipe. However, some things like like non-blocking IO on pipes isn't supported for anonymous pipes and changing the pipe buffer size is undocumented.

[milseman]

ftruncate as FileDescriptor.resize: https://github.com/apple/swift-system/pull/82

[ffried]

@milseman Would sendfile (or maybe rather copy_file_range) be a candidate that could make it to this package? If so, I'd be happy to create a PR.

[milseman]

@milseman Would sendfile (or maybe rather copy_file_range) be a candidate that could make it to this package? If so, I'd be happy to create a PR.

Yes, but I think that would be contingent on sockets, which got punted out (but hopefully we can readdress it soon)