Support ``F_SETFL`` and ``F_GETFL``

[tiif]

This is proposed to support some tests in tokio.

Ideally we could support F_SETFL and F_GETFL for all kind of file descriptions, but getting them to work on socketpair should be sufficient to unblock some of the tests there (here is the relevant codepath).

F_GETFL is quite straight forward as we only support SOCK_NONBLOCK and SOCK_CLOEXEC flags for socketpair.

For F_SETFL, as it has the ability to change O_NONBLOCK flag, we need to decide what should happen if a previously blocking fd is marked as non-blocking, and there are waiting threads. Should they wake up immediately, later at some point or never? It might make this previously unreachable case, reachable.

Ideally support this on:

• socketpair
• pipe

@rustbot label +A-shims +C-enhancement +A-concurrency

[RalfJung]

F_GETFL is quite straight forward as we only support SOCK_NONBLOCK and SOCK_CLOEXEC flags for socketpair.

Note that CLOEXEC is a file descriptor flag and not exposed via F_SETFL/F_GETFL. Those commands only handle file description flags. See the man page:

       Each open file description has certain associated status flags,
       initialized by open(2) and possibly modified by fcntl().
       Duplicated file descriptors (made with dup(2), fcntl(F_DUPFD),
       fork(2), etc.) refer to the same open file description, and thus
       share the same file status flags.

       The file status flags and their semantics are described in
       open(2).

       F_GETFL (void)
              Return (as the function result) the file access mode and
              the file status flags; arg is ignored.

       F_SETFL (int)
              Set the file status flags to the value specified by arg.
              File access mode (O_RDONLY, O_WRONLY, O_RDWR) and file
              creation flags (i.e., O_CREAT, O_EXCL, O_NOCTTY, O_TRUNC)
              in arg are ignored.  On Linux, this operation can change
              only the O_APPEND, O_ASYNC, O_DIRECT, O_NOATIME, and
              O_NONBLOCK flags.  It is not possible to change the
              O_DSYNC and O_SYNC flags; see BUGS, below.

[Darksonn]

Looking at the kernel source code, it seems that it does not notify existing blocking operations in any way, so I would assume that they just continue to block.

[RalfJung]

And then they get unblocked on the next read/write as usual? And at that point they might actually return EWOULDBLOCK if it turns out that not all threads could really be unblocked? I could imagine the kernel doing something smarter, but OTOH changing the blocking nature of a socket that has ongoing operations seems sufficiently odd that maybe we don't need to be bug-for-bug compatible with the kernel.

[Darksonn]

Yeah, not sure. I guess you'll have to try it if you want to find out.

[bjorn3]

For jobserver it would also be nice to support them on pipes.

[tiif]

I'd like to push this forward, looking into this soon.

@rustbot claim

[tiif]

I might be wrong, but I suspect that the kernel might race in reading / modifying the O_NONBLOCK flag.

In setfl, when the kernel is modifying a flag, a spinlock is used to protect the the flag.
https://github.com/torvalds/linux/blob/4dc1d1bec89864d8076e5ab314f86f46442bfb02/fs/fcntl.c#L82-L85

The comment here also says f_lock is used to protect f_flags. https://github.com/torvalds/linux/blob/4dc1d1bec89864d8076e5ab314f86f46442bfb02/include/linux/fs.h#L1071

But in pipe_read / pipe_write, the reading of flags is not protected by the same lock.
https://github.com/torvalds/linux/blob/4dc1d1bec89864d8076e5ab314f86f46442bfb02/fs/pipe.c#L357-L361

There could be other mechanisms that can prevent this from happening, but up until now, I still haven't found any.

If it'd be reasonable for us to define the behaviour, I would prefer to make fcntl throws EAGAIN when there is 1 or more threads blocking on the target fd (for example, socketpair / pipe).

[RalfJung]

make fcntl throws EAGAIN when there is more than 1 threads blocking on it.

Does fcntl ever return that error on real systems? If not, then I don't think it would be a good idea to do that in Miri.

[tiif]

Does fcntl ever return that error on real systems?

Yes, these are documented in https://man7.org/linux/man-pages/man2/fcntl.2.html#ERRORS

       EACCES or EAGAIN
              Operation is prohibited by locks held by other processes.

       EAGAIN The operation is prohibited because the file has been
              memory-mapped by another process.

The reason why I chose EAGAIN is because the description is the closest to the scenario we have here.

       EAGAIN Resource temporarily unavailable (may be the same value as
              EWOULDBLOCK) (POSIX.1-2001).

the above is from taken from https://man7.org/linux/man-pages/man3/errno.3.html