rework the POSIX API layer in the standard library

[andrewrk]

Currently, we have these OS-specific APIs:

• std.os.linux
• std.os.windows
• std.os.darwin
• std.os.freebsd
• std.os.netbsd
• std.os.wasi

std.os.posix is defined like this:

pub const posix = switch (builtin.os) {
    Os.linux => linux,
    Os.macosx, Os.ios => darwin,
    Os.freebsd => freebsd,
    Os.netbsd => netbsd,
    Os.wasi => wasi,
    else => @compileError("Unsupported OS"),
};

So what we have is an identifier posix that is mapped directly to the OS-specific API for all platforms except Windows. This is problematic for several reasons:

• Each OS-specific API has a different set of functions available.
• Some functions have different number of or incompatible parameters.
• It's quite possible to implement a POSIX API layer on top of windows, at least for some things.
• We have this std.os.posixFoo pattern, which is a code smell, because all the functions are all prefixed with posix but cannot be in the posix namespace.
• There is a need for "glue code" to work around kernel API footguns. For example when the number of iov_len is greater than some kernel-defined limit, it has to be broken into multiple syscalls. Or when the number of bytes to the write syscall is greater than some number, it has to be broken into multiple syscalls.

This led to 4a8c992 in which, for example, std.os.linux.fork surprisingly does not cause a fork syscall. Further, with the WASI ABI, it's clear that there is a separate "core" API layer as well as a "POSIX" API layer.

I propose to make the following changes:

• Make the OS-specific files do exactly what they say. E.g. if you call std.os.linux.fork, that is going to do the fork syscall when you observe the program with strace, guaranteed.
• Instead of std.os.posix mapping directly to OS-specific files, it will provide a "zig flavored POSIX" API on top of the native OS API. This is where kernel limitations will be worked around; fork might call clone on Linux, etc.
• Functions matching the pattern std.os.posixFoo will be moved to std.os.posix.foo. In Zig-flavored POSIX, errno is not exposed; instead actual zig error unions and error sets are used. When not linking libc on Linux there will never be an errno variable, because the syscall return code contains the error code in it. However for OS-specific APIs, where that OS requires the use of libc, errno may be exposed, for example as std.os.darwin.errno().
• There will even be a zig flavored POSIX API for Windows. Some functions may cause a compile error if the abstraction does not hold up, but for some things, it will work fine.

[daurnimator]

How far should the "posix" library go? Only include things specified by posix? What about APIs that almost all posix platforms provide?

I imagine the file structure will be:

std/os/linux.zig
...
std/os/freebsd.zig
std/posix.zig
std/posix/linux.zig
...
std/posix/freebsd.zig

Where std/posix.zig dispatches the relevant OS-specific wrappers?

[andrewrk]

How far should the "posix" library go? Only include things specified by posix?

It can have everything that would be convenient to abstract across operating systems at the posix layer. It can go beyond posix specifications. For example there can be a kqueue function. The goal is not to be compliant in any sense; more to provide an API at the same level of abstraction.

What about APIs that almost all posix platforms provide?

They're in 👍

I imagine the file structure will be:

That looks right. The std/posix/linux.zig etc files might not be necessary, depending on how much code there ends up being in std/posix.zig.

[daurnimator]

It can have everything that would be convenient to abstract across operating systems at the posix layer. It can go beyond posix specifications. For example there can be a kqueue function. The goal is not to be compliant in any sense; more to provide an API at the same level of abstraction.

I think using "posix" to mean anything other than what is actually defined as API/ABI in the posix standard itself is dangerous: what happens if POSIX gets updated and gains functions with the same name as whatever we define?

Perhaps we should name our module something else? And have a "posix" module only contain what is actually in posix? That way if your program only used the posix library, then you'd know that your zig program is portable to any OS that claims POSIX compliance (which of course is the whole point of POSIX).

[hryx]

Hijacking the word "posix" does feel weird. Why not have cross-OS abstractions just live in std/os?

[mikdusan]

If std.os.posix is to be convenient and tailored to Zig, that sounds very similar to std.os. In other words, why not just have std.os.kqueue instead of std.os.posix.kqueue?

My main question is what would differentiate std.os vs. std.os.posix? Is it to mean std.os.posix is intended to be "lower-level" API than std.os?

[andrewrk]

Why not have cross-OS abstractions just live in std/os?

We do have those already, and those are the preferred cross-platform abstractions for general use. Most of the implementations of these use the posix API layer for non-windows. This abstraction layer exists today. If it's not called posix then it has to be called something. Right now it's awkwardly called std.os.posixFoo (where Foo is the posix function name). Have a look at std/os.zig to see what I mean. This proposal would strictly be an improvement, even if everybody agreed that "posix" was a bad name for this abstraction layer.

An example "posix" abstraction layer function is to write to a file descriptor. That's too low-level for std.os. An example std.os abstraction layer function is std.os.File.open and std.os.File.write. The File struct is a cross platform abstraction that removes the concept of file descriptors.

what happens if POSIX gets updated and gains functions with the same name as whatever we define?

If POSIX got a sendmmsg then it would work perfectly. What other example are you thinking of that is problematic?

[daurnimator]

If POSIX got a sendmmsg then it would work perfectly. What other example are you thinking of that is problematic?

There have been plenty of times in the past where things have gotten mixed up as they make their way into posix.
A recent example of an almost-posix API that I've had trouble with is pthread_setname_np. It is slightly different on every posix system; and each with their own caveats. See wahern/cqueues#208 (comment)

[hryx]

[std.os] are the preferred cross-platform abstractions for general use

Ah, so this new "posix" is more of an abstraction for the sake of stdlib internals? (But possibly also useful outside of that?)

This abstraction layer exists today.

Right — sorry, I didn't quite get that before. Like this?
https://github.com/ziglang/zig/blob/master/std/os/time.zig#L31

I think it makes more sense to me with that explanation.

[andrewrk]

Ah, so this new "posix" is more of an abstraction for the sake of stdlib internals? (But possibly also useful outside of that?)

Yes, precisely. std.os.posixSleep that you linked is a good example.