diff --git a/libc/docs/abi_policy.md b/libc/docs/abi_policy.md new file mode 100644 index 0000000000000..a2685caa33ccb --- /dev/null +++ b/libc/docs/abi_policy.md @@ -0,0 +1,186 @@ +(abi_policy)= + +# ABI and Public Header Policy + +LLVM-libc defines its own ABI. This page explains how the layout of public +types is chosen, which symbols the library exports, and how public headers are +written. It applies to full builds, where LLVM-libc provides the headers. In +{ref}`overlay mode ` the headers and the ABI belong to the system +libc. + +See {ref}`design_philosophy` for why the project works this way. + +## Nothing is frozen + +LLVM-libc currently makes no ABI stability promise. The layouts of public +types, the names of internal symbols, and the values of implementation-defined +macros may change between two commits. Because the library is +{ref}`linked statically into the program it was built for `, +such a change does not break existing binaries. + +This lack of stability gives the project the freedom to change the ABI without +fear of breaking users. If there's a mistake in the ABI, it can be fixed quickly +and uncontroversially. + +The project intends to keep this freedom for as long as possible. A change that +would make part of the ABI hard to change later needs discussion with the +maintainers before it is written. + +(choosing_type_layout)= + +## Choosing the layout of a public type + +When a public type is added or changed, its layout is decided by the first of +these that applies: + +1. **The kernel.** If the type is passed to or received from a system call + without conversion, it must have the layout the kernel expects. `struct + timespec` and `stack_t` are examples. +2. **LLVM-libc.** Otherwise, design the type for LLVM-libc's implementation, + and convert at the system call boundary where necessary. `struct stat` is + LLVM-libc's own layout and is filled in from `statx` + (`src/__support/OSUtil/linux/stat/`). `struct sigaction` is converted to and + from the kernel's layout (`KernelSigaction` in + `src/signal/linux/signal_utils.h`). +3. **An existing implementation**, only when there is no other source of + truth. This happens when the kernel's UAPI headers are wrong or unusable + for user space, or when an interface has no real standard and one + implementation has become the de facto definition, as glibc has for + `termios`. + +"A program built against another libc's headers expects this layout" is not a +reason to choose that layout. Programs that rely on details a standard does not +promise should be fixed, unless no standard exists at all. + +### Opaque types + +Where a standard allows a type to be opaque, keep it opaque. As an example, +LLVM-libc's `pthread_t` is a struct, which POSIX explicitly permits. Keeping it +a struct means its contents can change later without breaking correct programs. +Correct code that needs to compare thread IDs uses `pthread_equal`. + +`FILE` and `DIR` are forward declarations in the public headers. Nothing +outside the library can depend on their contents. + +### Widths of common types + +In general it's best to have a single definition of a given type. A type that +differs between platforms requires extra care wherever it is handled. The `long` +type is an example: it is 32 or 64 bits wide depending on the target. `long` is +defined by the platform and the compiler, so LLVM-libc has to handle both +widths and cannot standardize it. + +For types that are provided by LLVM-libc, it's generally preferred to use the +larger type. Smaller values will fit in larger types without issue, but the +reverse is not true. + +`time_t` is 64 bits on every target, because 32-bit time ends in 2038. The +types that travel with it, such as `off_t`, `dev_t`, and `rlim_t`, are also 64 +bits everywhere. This simplifies the build, and it means the `*64` variants of +file functions are not needed. + +A few related types (`ino_t`, `nlink_t`, `blkcnt_t`) are still pointer width. +That is unfinished work and not a design decision. + +## Feature-test macros + +Public headers do not use feature-test macros such as `_GNU_SOURCE` or +`_POSIX_C_SOURCE` to hide declarations. Do not add `#ifdef` guards of that kind +to a header. The exception is a macro that a standard itself defines for this +purpose, such as `__STDC_WANT_LIB_EXT1__` for Annex K. + +Headers are generated from the YAML files in `libc/include/` (see +{ref}`header_generation`), and a generated header declares only the functions +that are enabled for the target. A user who wants a different API surface +generates headers to match it, by pointing the `LIBC_CONFIG_PATH` CMake option +at their own `entrypoints.txt`, `headers.txt`, and configuration (see +{ref}`custom_config_directory`), instead of every user receiving one set of +headers full of conditionals. + +Each function in the YAML files lists the `standards` it comes from (`stdc`, +`posix`, `gnu`, `bsd`, `linux`, `llvm_libc_ext`, and so on). Keep that field +accurate. It records where each declaration comes from. + +## Exported symbols + +Every public function is defined in the internal `LIBC_NAMESPACE` namespace and +exported under its standard C name by the `LLVM_LIBC_FUNCTION` macro (see +{ref}`implementation_standard`). The default namespace name includes the LLVM +version, and {ref}`Hand-in-Hand ` consumers compile the shared +code under a different namespace, so several copies of LLVM-libc code can be +linked into one program. Nothing may depend on the mangled names of internal +symbols. + +### Aliases for another libc's symbols + +LLVM-libc does not export another libc's private symbols in general. + +The exception is the `LIBC_COPT_PROVIDE_GNU_ALIASES` option, which is enabled +for Linux. It provides a short list of glibc symbol names, such as +`__errno_location`, because some users must link precompiled libraries that +were built against glibc. The list is kept short deliberately. It is a +compromise for those users and not a step toward binary compatibility with +glibc. Additions need agreement from the maintainers. + +Symbols that a compiler or a C++ standard library requires of any libc, such +as `__stack_chk_fail` or `__cxa_thread_atexit_impl`, are ordinary entrypoints +and are not covered by this restriction. + +### Customization points + +Some macros exist so that downstream users can control how LLVM-libc code is +compiled into their own library. Downstream users depend on `LIBC_INLINE` and +`LIBC_INLINE_VAR` being used on every inline function and variable, and +`LLVM_LIBC_FUNCTION_ATTR` lets a packager attach attributes to every public +function. Use these macros as documented in {ref}`code_style`, and do not +remove or bypass them. + +## Kernel headers + +A full build for Linux requires the kernel's UAPI headers. + +A public header may include a UAPI header when that header is self-contained +and does not collide with LLVM-libc's own definitions. When a UAPI header +redefines a type LLVM-libc provides, or defines macros with different +spellings, define the needed constants and types in `llvm-libc-macros/` or +`llvm-libc-types/` instead. For example, UAPI headers that include +`` redefine `struct flock` and so cannot be included. + +## Dynamic loading + +LLVM-libc is linked statically today. Dynamic loading is under development, +mainly so that `dlopen` can work. It is not intended to become the primary way +to use the library, and new code should be designed for the static case first. + +## Other operating systems + +The points above describe Linux and baremetal targets, where LLVM-libc is the +whole C library. A full build for macOS or Windows has not been designed yet. +The likely direction is for LLVM-libc to define its own API there too, shaped +appropriately for the platform, and not to reproduce POSIX on every system. + +## Writing public headers + +Public headers are read by every program and by compilers in every language +mode, so they are held to stricter rules than implementation code: + +- A header must be valid in every C and C++ standard mode. Do not use language + features that some of those modes lack, such as attribute syntax (`[[...]]`), + which C gained only in C23. +- Names that the standard does not define must be reserved identifiers. This + includes structure members, macro parameters, and helper macros. A member + named `mutex` or a macro named `NUM_LOCALE_CATEGORIES` collides with user + code. +- Type headers in `llvm-libc-types/` define one type each and include only the + other type headers they need. Use the compiler's built-in type macros (for + example `__INT64_TYPE__`) instead of including ``. + +## Proposing an ABI change + +To propose a non-trivial change to the layout of an existing public type or +the value of an existing public macro, or to export a symbol under a name +other than its standard one, contributors should start by reading +{ref}`choosing_type_layout`, then reach out to the maintainers with the +justification and/or write an RFC with the new design. A demo PR may be +helpful for explaining the design, but should not be expected to land without +significant changes. diff --git a/libc/docs/arch_support.md b/libc/docs/arch_support.md index a9b389e2ca91e..8177579559cd8 100644 --- a/libc/docs/arch_support.md +++ b/libc/docs/arch_support.md @@ -1,16 +1,28 @@ # Architecture Support -The currently continuously tested architectures are: +The table below lists what is built and tested on every pull request. -- aarch64 -- amdgpu -- arm -- nvptx -- riscv32 -- riscv64 -- x86_64 +| Architecture | Tests run | Built only | +| --- | --- | --- | +| x86-64 | Linux (overlay and full build), Windows (overlay), FreeBSD (overlay) | UEFI | +| AArch64 | Linux (overlay and full build), macOS (overlay) | | +| 32-bit Arm | Linux (overlay, under QEMU) | Baremetal | +| 64-bit RISC-V | Linux (overlay, under QEMU) | | +| 32-bit RISC-V | | Baremetal | +| Hexagon | Baremetal | | +| AMDGPU | | GPU | -i386 support is [in the works](https://github.com/llvm/llvm-project/issues/93709). +NVPTX is also a supported GPU target. See {doc}`the GPU docs ` for +both GPU targets. + +After a change lands, the +[postcommit buildbots](https://lab.llvm.org/buildbot/#/builders?tags=libc) +currently cover Linux on x86-64: Clang and GCC, overlay and full build, and +AddressSanitizer. + +i386 support is [in the works](https://github.com/llvm/llvm-project/issues/93709), +though it may be sunset. See {ref}`platform_support` for the status of each +target. See {doc}`Bringup on a New OS or Architecture ` for more information. Please do first file a bug in diff --git a/libc/docs/build_and_test.md b/libc/docs/build_and_test.md index 7c33fb9a445d2..fe7598f976016 100644 --- a/libc/docs/build_and_test.md +++ b/libc/docs/build_and_test.md @@ -8,39 +8,44 @@ The libc can be built and tested in two different modes: 1. **The overlay mode** - In this mode, one uses the static archive from LLVM's libc along with the system libc. See {ref}`overlay_mode` for more details - on building and using the libc in this mode. You can only run the libc - unittests in this mode. To run them, one simply does: + on building and using the libc in this mode. To run the tests, one simply + does: ```sh $> ninja check-libc ``` - Note that, unittests for only those functions which are part of the overlay + Note that, tests for only those functions which are part of the overlay static archive will be run with the above command. 2. **The full build mode** - In this mode, the libc is used as the only libc for the user's application. See {ref}`full_host_build` for more details on building and using the libc in this mode. Once configured for a full libc - build, you can run three kinds of tests: + build, the same command builds the tests against LLVM-libc's own headers + and startup code and runs them: - 1. Unit tests - You can run unittests by the command: + ```sh + $> ninja check-libc + ``` - ```sh - $> ninja check-libc - ``` + Subsets of the tests have their own targets: - 2. Integration tests - You can run integration tests by the command: + 1. Integration tests - You can run integration tests by the command: ```sh $> ninja libc-integration-tests ``` - 3. Shared tests - You can run tests for shared, standalone components (like math primitives) without needing the full libc runtime by the command: + 2. Shared tests - You can run tests for shared, standalone components (like math primitives) without needing the full libc runtime by the command: ```sh $> ninja libc-shared-tests ``` +A change should be tested in both modes. See {ref}`testing_guide` for the +kinds of tests, how to run a single test, and how to confirm that a test was +not silently skipped. + ## Building with VSCode As a quickstart to using VSCode for development, install the cmake extension diff --git a/libc/docs/compiler_support.md b/libc/docs/compiler_support.md index 790541f9d4290..1c41c83eebc08 100644 --- a/libc/docs/compiler_support.md +++ b/libc/docs/compiler_support.md @@ -17,3 +17,9 @@ is required it is possible to bootstrap `Clang` with `GCC` and then use - `Clang 11` - `GCC 12.2` + +Code that is shared with other LLVM projects through +{ref}`Hand-in-Hand ` is compiled as part of those projects, so +it must also build with the +[host compilers LLVM supports](https://llvm.org/docs/GettingStarted.html#host-c-toolchain-both-compiler-and-standard-library), +which are older. diff --git a/libc/docs/contributing.md b/libc/docs/contributing.md index 8e53f8f5501b9..d2420186a3e09 100644 --- a/libc/docs/contributing.md +++ b/libc/docs/contributing.md @@ -13,7 +13,9 @@ a list of open projects that one can start with: issue" by the team as potentially easier places to get started. Please do first check if the bug has an assignee; if so please find another unless there's been no movement on the issue from the assignee, in which place do - ask if you can help take over. + ask if you can help take over. Remember to follow the LLVM + [AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html) which forbids + using LLMs for good first issues. 2. **Cleanup code-style** - The libc project follows the general [LLVM style](https://llvm.org/docs/CodingStandards.html) with specific conventions for naming (`snake_case` for functions, `CamelCase` for @@ -39,3 +41,72 @@ a list of open projects that one can start with: 6. **Contribute a new OS/Architecture port** - You can contribute a new operating system or target architecture port. See {ref}`porting` for more information. + +## Before you start + +Read {ref}`design_philosophy` and {ref}`abi_policy` to understand what the +project is and is not trying to do. Read {ref}`patterns_to_avoid` before +modelling new code on something that is already in the tree, because older +code does not always follow the current conventions. + +Some changes are worth discussing with the maintainers before investing in an +implementation: + +- a non-trivial change to the layout of an existing public type or the value + of an existing public macro, or exporting a symbol under a name other than + its standard one (see {ref}`abi_policy`), +- a function that is not in the C or POSIX standards, +- a new configuration option, +- new code under `libc/shared/`, or a new user of it, +- any code written in assembly, +- any intentional deviation from a standard. + +The easiest way to reach out is to ask in the `#libc` channel on the +[LLVM Discord](https://discord.gg/xS7Z362). + +## Pull requests + +- **Keep pull requests small and focused.** Add one function, or one closely + related family of functions, per pull request. Land refactoring separately + from the change that needs it. A bug found along the way gets its own pull + request. +- **Title.** Start the title with `[libc]`, and add a second tag where one + applies: `[libc][math]`, `[libc][docs]`, or `[libc][NFC]` for changes with + no functional effect. +- **Format the change.** Run `clang-format` on C++ sources and + `libc/utils/cmake_format.py -i` on the CMake files you edited. +- **Test in both build modes** where the code supports both. See + {ref}`testing_guide`. +- **Do not mention people with `@` in commit messages**, or in a pull request + description that will become one. GitHub notifies them again each time the + commit is pushed to another fork. Request reviews through the reviewers + field instead. +- **Bazel.** Updating the Bazel build files is not required, but it is + appreciated. + +LLVM-libc follows LLVM's +[patch reversion policy](https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy). +A change that breaks a buildbot is reverted or, if the fix is trivial, fixed +forward. When relanding, describe what went wrong and how it was fixed. + +## AI-assisted contributions + +LLVM's [AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html) applies +to LLVM-libc. In short: a person must read, understand, and be able to answer +questions about everything they submit; contributions with substantial +tool-generated content are labelled, for example with an `Assisted-by:` +trailer; the policy strongly recommends writing the pull request description +yourself; and AI tools must not be used on issues labelled "good first issue". + +Tools make some mistakes in this project more often than people do: + +- They imitate the code around them, including code that follows outdated + conventions. See {ref}`patterns_to_avoid`. +- Asked to make a program work, they change LLVM-libc's types, symbols, or + behavior to match another libc, because that is the shortest path to a + passing build. See {ref}`making_existing_software_work`. +- They reproduce code from other C libraries. That is not allowed, regardless + of how the code was produced. +- They call one public function from another. See {ref}`code_organization`. + +Check for these before asking for review. diff --git a/libc/docs/design_philosophy.md b/libc/docs/design_philosophy.md new file mode 100644 index 0000000000000..656ef741a7327 --- /dev/null +++ b/libc/docs/design_philosophy.md @@ -0,0 +1,175 @@ +(design_philosophy)= + +# Design Philosophy + +This page describes what LLVM-libc is for, who it serves, and what it +deliberately does not try to be. Most of the rules elsewhere in the developer +documentation follow from the decisions recorded here. Read this page before +starting a change whose goal is "make program X work with LLVM-libc", because +the easiest way to reach that goal is often the wrong one for the project. + +## An application libc + +Most C libraries are *system* libcs. One shared copy is installed on a machine, +every program on that machine uses it, and so its ABI and observable behavior +are very hard to change. LLVM-libc is designed as an *application* libc: it is +built for a specific program and linked into it. + +This has consequences throughout the project: + +- **Static linking is the primary use case.** Dynamic loading is under + development because features such as `dlopen` need it, but it is not the + primary use case and should not drive design decisions. +- **There is no stable ABI.** The library and the program are built together, + so type layouts and internal symbols are free to change. See + {ref}`abi_policy`. +- **The library is configured for its user.** Each target selects its own set + of entrypoints, configuration options, and generated headers, instead of the + library shipping one build that tries to suit everyone. +- **Users track `main`.** There is no separate release stream for LLVM-libc + today. This is likely to change when the library is shipped more widely. + +Writing a new libc is an opportunity to design an API and ABI for modern +systems. Preserving the freedom to do that is worth more to the project than +making any one program work sooner. + +## Who we build for + +LLVM-libc has several kinds of users: statically linked programs on Linux +servers, baremetal embedded systems, GPUs, other LLVM projects that use its +internals through {ref}`Hand-in-Hand `, and general open-source +software. Their needs differ. In general, baremetal users prioritize binary +size, for example, while server users prioritize latency and throughput. + +The first choice is always a design that serves every user. When that is not +possible, a feature added for one group of users must not impose a cost on +the others that they cannot configure away. + +The longer-term milestones for general software support are building web +browsers (Chromium and Firefox) and the generic Linux utilities needed for a +complete Linux system built with LLVM and no GNU components. + +## Non-goals + +- **Compatibility with glibc, or any other libc, as a goal in itself.** + LLVM-libc implements the C and POSIX standards. It does not pretend to be + another implementation. See {ref}`making_existing_software_work` for what + this means in practice. +- **Bug-for-bug compatibility.** Where a standard defines the behavior, + LLVM-libc follows the standard even if a widely used libc does not. See + {ref}`undefined_behavior`. +- **32-bit `time_t`.** `time_t` is 64 bits wide on every target. +- **"POSIX everywhere".** On platforms that are not POSIX systems, the API + should be shaped for the target and not be an emulation of Linux. + +The following are *not* non-goals. They are missing or incomplete today: + +| Feature | Status | +| --- | --- | +| Dynamic loading | Under development. Never intended to be the primary use case. | +| Time zone database | Planned. The intent is to base the implementation on the one in libc++. | +| Thread cancellation | Low priority. When added, it must not significantly slow down code that does not use it. | +| Message catalogs, Annex K | Low priority. | +| Locales | See the next section. | + +## Text, encodings, and locales + +The locale design in the C standard conflates two different kinds of text: +text exchanged with other programs (file formats, protocols, command lines) and +text shown to or read from a person. Modern software needs to treat these +differently, and a process-wide locale that silently changes the behavior of +`printf` or `strtod` makes the first kind unreliable. + +LLVM-libc's direction is: + +- **Functions without an explicit locale argument ignore locales.** They + assume their input or output is exchanged with another program. +- **`char` holds the platform's native encoding.** That is usually ASCII + compatible, but it may be EBCDIC: libc++ supports IBM targets, and LLVM-libc + supports libc++. Code must not assume ASCII values or that letters are + contiguous. +- **Multibyte strings are UTF-8.** +- **`wchar_t` strings are UTF-32 or UTF-16**, depending on the target. UTF-16 + support is in progress. +- **Encoding assumptions are centralized.** They live in + `src/__support/ctype_utils.h`, `src/__support/wctype_utils.h`, and the + converters in `src/__support/wchar/`. All code uses those helpers instead of + doing arithmetic on characters, including greater/less than comparisons. +- **Text for people goes through explicit-locale functions.** The intent is to + provide the `_l` variants of the relevant functions, so that a caller who is + formatting text for display passes a locale explicitly. + +Today only the "C" locale is implemented. A function that depends on a locale +implements the "C" locale behavior and does not claim to do more. + +(making_existing_software_work)= + +## Making existing software work + +Building real programs against LLVM-libc is how missing pieces are found, and +those reports are welcome. What matters is how each gap is closed. In order of +preference: + +1. **Fix LLVM-libc where it deviates from a standard.** A wrong return value + or a missing error condition is simply a bug. +2. **Implement the missing standard function, macro, type, or header.** +3. **Add transitive includes in public headers that POSIX requires** when a + real program relies on them. +4. **Add a widely used extension** (GNU, BSD, or Linux specific) when a real + program needs it. See {ref}`extensions_and_stubs`. +5. **Accommodate a non-standard assumption** only when doing so is cheap and + does not limit future changes to LLVM-libc. Declaring `alloca` from + `` is an example of an acceptable accommodation. +6. **Change the program, or how it is configured,** when it depends on the + internals of another libc or on behavior no standard promises. + +The following are never acceptable ways to make a program work: + +- Changing the layout or kind of a public type solely to match another libc. For + example, `pthread_t` is a structure in LLVM-libc. POSIX allows that, and code + that treats `pthread_t` as an integer or a pointer is not portable. Making it + a scalar would remove the freedom to change it later. +- Changing observable behavior to match another libc's deviation from a + standard. +- Adding another libc's private symbols so that objects compiled against its + headers link. The one narrow exception is described in {ref}`abi_policy`. + +If a task can only be completed by doing one of these, stop and raise it with +the maintainers instead. + +(extensions_and_stubs)= + +## Extensions, stubs, and experimental entrypoints + +Before implementing a function that is not in a standard, ask whether anyone +needs it. An extension needs a specific user: a named program or library that +calls it. + +A function that is declared and linkable but not fully implemented goes behind +the `LLVM_LIBC_ENABLE_EXPERIMENTAL_ENTRYPOINTS` CMake option. The bar for +adding a stub there is low, with two conditions: + +- It has a specific user, as above. +- A call that the stub cannot handle fails clearly. A stub that silently + reports success hides the missing functionality from the people who would + otherwise report or fix it. + +An entrypoint leaves the experimental list when it implements all of the +functionality its specification requires. Some incomplete entrypoints predate +the option and are still in the default lists; they should move. + +## Original implementations only + +Do not copy code from other libc implementations, and do not port it line by +line. This is a licensing requirement, and it is also a question of style: +LLVM-libc is written in idiomatic C++, while most other libcs are written in +C. Code translated from C rarely fits the conventions described in the +{ref}`developer guides `. + +Standards documents, manual pages, and observing the behavior of another +implementation as a black box are all appropriate references. + +If you own an existing implementation of a libc function that you would like to +contribute, that is likely fine assuming it is updated to match the LLVM-libc +style and placed under the LLVM license. Reach out to the maintainers for more +information. diff --git a/libc/docs/dev/building_docs.md b/libc/docs/dev/building_docs.md index 796f5d096f5c0..ca254b59aa043 100644 --- a/libc/docs/dev/building_docs.md +++ b/libc/docs/dev/building_docs.md @@ -79,8 +79,8 @@ changes will be overwritten the next time the docs are built. Otherwise: `pip install -r llvm/docs/requirements.txt`. `WARNING: document isn't included in any toctree` -: A new RST/Markdown file needs a `toctree` entry. Add it to the - appropriate `index.rst` or its parent toctree. +: A new Markdown file needs a `toctree` entry. Add it to the + appropriate `index.md` or its parent toctree. `Extension error: No module named 'sphinx_reredirects'` : Same fix: `pip install -r llvm/docs/requirements.txt`. diff --git a/libc/docs/dev/code_organization.md b/libc/docs/dev/code_organization.md new file mode 100644 index 0000000000000..b0ab8044f59b5 --- /dev/null +++ b/libc/docs/dev/code_organization.md @@ -0,0 +1,200 @@ +(code_organization)= + +# Where Code Goes + +This page explains how to decide where a piece of code belongs: in an +entrypoint, in a helper next to it, in `src/__support/`, or in `shared/`. It +also covers which build modes and targets a new function is enabled for. +{ref}`source_tree_layout` describes the directories themselves. + +## Three kinds of code + +It helps to sort code in a libc into three kinds: + +- **OS independent**: pure computation with no system interaction, such as + the math and string functions. +- **OS agnostic**: standard interfaces whose behavior is the same everywhere + but whose implementation needs the system, such as `fopen` or `assert`. +- **OS specific**: interfaces that are specific to a platform. This can be a + thin interface like a syscall wrapper or a substantial integration like + `popen`. + +The source tree does not separate these three as cleanly as it should, and +moving toward that separation is a long-term goal. When adding code, decide +which kind it is and keep OS-specific details out of the first two. + +## Entrypoints do not call entrypoints + +An entrypoint is the definition of one public function. It may contain its +whole implementation. As soon as any part of that implementation is needed +somewhere else, that part moves into a separate internal header or library. + +**An entrypoint must not call another entrypoint.** This is the most common +mistake in new code. Calling `strlen` from `strdup`, or `fseek` from +`fsetpos`, is the natural way to write the function. It is wrong here because +every entrypoint is a separate public symbol in its own separate object file +that each target enables or leaves out individually. A dependency between two +entrypoints means that they must be pulled in together. + +Instead, put the shared logic in an internal function and call it from both +entrypoints: + +```c++ +// src/string/string_length.h +namespace LIBC_NAMESPACE_DECL { +namespace internal { +LIBC_INLINE size_t string_length(const char *src) { ... } +} // namespace internal +} // namespace LIBC_NAMESPACE_DECL + +// src/string/strlen.cpp +LLVM_LIBC_FUNCTION(size_t, strlen, (const char *src)) { + return internal::string_length(src); +} +``` + +CMake checks for this. When configured with `-DLIBC_CMAKE_VERBOSE_LOGGING=ON`, +it warns about any entrypoint or internal library that depends on an +entrypoint. The few permitted exceptions, currently `errno` and `longjmp`, are +listed in `check_entrypoint_deps` in +`libc/cmake/modules/LLVMLibCObjectRules.cmake`. + +Internal libraries may depend on *variable* entrypoints such as `environ` or +`stdout` when they need the object itself. They may not depend on function +entrypoints. + +## Where shared logic lives + +| The logic is used by | Put it in | +| --- | --- | +| One entrypoint | The entrypoint's `.cpp` file, as `static` functions. | +| A few entrypoints, normally from the same public header | A header or object library in that header's directory, for example `src/stdio/fflush_internal.h`, in the `internal` namespace. | +| Many parts of the library, or other LLVM projects | `src/__support/`. | + +Avoid creating new files or directories named `utils`. A name that describes +nothing attracts unrelated code. Name a helper after what it does. + +Keep internal interfaces as narrow and as typed as public ones: + +- Functions whose result becomes `errno` return `ErrorOr` and never set + `errno` themselves. See {ref}`setting_errno`. +- Reuse an existing parser or type before writing a narrower one. For example, + `stdio` mode strings are parsed by `FileMode`. +- Write validity checks positively ("is one of the supported values"), not as + a list of rejected cases. +- Prefer a fixed signature over an unconstrained template when the body only + works for specific types. + +## Code shared with other LLVM projects + +Headers in `libc/shared/` expose LLVM-libc internals to other LLVM projects +through {ref}`Hand-in-Hand `. + +`shared/` is only an interface. The implementation lives under `libc/src/`, +usually in `src/__support/`, and the header in `shared/` re-exports it in the +`shared` namespace. Code inside LLVM-libc includes the implementation from +`src/` and does not include anything from `shared/`. The GPU RPC headers are +an existing exception: they are implemented in `shared/` itself. + +Everything reachable from `shared/` has extra constraints: + +- It is header-only. +- It does not depend on any external libc function. The allocator is the one + exception, because it is external to LLVM-libc on most targets already. +- It builds with the compilers LLVM itself supports, which are older than the + minimum for the rest of LLVM-libc. See {ref}`compiler_support`. + +This covers the `printf` core, string-to-number conversion, the math +implementations in `src/__support/math/`, and what they include from +`src/__support/`. Adding a new header to `shared/`, or a new consumer of it, +should be discussed with the maintainers first. + +## Operating-system-specific code + +Per-OS duplication is acceptable. Operating systems differ enough that sharing +an implementation between them usually costs more than it saves. + +System calls are made only from wrappers in `src/__support/OSUtil//`; see +{ref}`syscall_wrapper_refactor`. OS-agnostic code never names an OS-specific +namespace such as `linux_syscalls`. + +There are several ways to select the implementation for the target OS. Pick the +one that fits the amount of code that differs: + +- **The whole entrypoint differs.** Put each implementation in + `src/
//.cpp`, and make the entrypoint in + `src/
/CMakeLists.txt` an `ALIAS` that depends on + `.${LIBC_TARGET_OS}.`. Most syscall-backed functions work this way. +- **The entrypoint is common and one step differs.** Keep a single + `src/
/.cpp` that calls an `internal::` function, and provide + that function as a per-OS object library selected with the same `ALIAS` + mechanism. `fflush` and its `fflush_internal` library are the example. +- **A small header-only primitive differs.** Use a dispatch header that + includes the right implementation, as `src/__support/OSUtil/io.h` does. + +## Overlay mode and full build + +A function may be enabled in {ref}`overlay mode ` only if it +does not depend on an implementation-defined type or on library-owned state +that is shared between functions. For example: + +- `strlen` and `round` are pure functions, so they are fine in overlay mode. +- `fopen` and `fgetc` are full build only, because `FILE` is implementation + defined. +- `atexit` is full build only, because it sets state that `exit` must read. +- `strtok` is fine in overlay mode. It keeps internal state, but no other + function reads that state. + +In each `entrypoints.txt`, overlay-safe functions are listed unconditionally and +the others are inside `if(LLVM_LIBC_FULL_BUILD)`. + +Both modes matter. Overlay mode is the more portable of the two and is how many +users adopt the math and memory functions; a full build is where everything +else is exercised. A change must build and pass its tests in both modes unless +the code is full-build only. See {ref}`testing_guide`. + +## Which targets get a new function + +A new entrypoint should be enabled on every target where it works. In practice +contributors cannot test every target, so the minimum is: + +- For a function that needs Linux, the `entrypoints.txt` files for `x86_64`, + `aarch64`, `arm`, and `riscv` under `libc/config/linux/`. The Linux + configuration for 32-bit Arm is also how that architecture is tested; see + {ref}`platform_support`. +- For an OS-independent function, the Linux configurations above plus the + baremetal, GPU, and other OS configurations, as described for math functions + in `libc/src/math/docs/add_math_function.md`. + +`linux/x86_64` is the most complete configuration. An entrypoint that is +present there and missing from another Linux architecture is usually an +oversight, not a decision, and adding it is a welcome change. + +The public header declaration comes from `libc/include/
.yaml`. A +function that is missing there will build and pass the tests but will not appear +in the generated header. + +### Experimental entrypoints + +Entrypoints with known-incomplete implementations are listed inside +`if(LLVM_LIBC_ENABLE_EXPERIMENTAL_ENTRYPOINTS)` and are off by default. See +{ref}`extensions_and_stubs` for when a function belongs there. + +## Configuration options + +Options in `libc/config/config.json` are named `LIBC_CONF_*` and are CMake +settings. They become `LIBC_COPT_*` macros on the compiler command line. See +{ref}`configure_options`. + +Add an option when a standard requires one behavior and a real user needs +another. Disabling floating point support in `printf` for small targets is the +model case. Do not add an option to avoid making a design decision. + +An option that disables behavior in an entrypoint must disable the matching +checks in that entrypoint's tests under the same condition. + +## Bazel + +The Bazel build in `utils/bazel/llvm-project-overlay/libc/` is maintained by +its downstream users, and an automated process keeps it in sync. Upstream +contributors are not required to update it, but it is appreciated. diff --git a/libc/docs/dev/code_style.md b/libc/docs/dev/code_style.md index f831c16bad617..7fac2b618ac49 100644 --- a/libc/docs/dev/code_style.md +++ b/libc/docs/dev/code_style.md @@ -250,7 +250,7 @@ The `include/` directory contains public facing headers that users must consume for fullbuild mode. As such, types defined here will have ABI implications as these definitions may differ from the underlying system for overlay mode and are NEVER appropriate to include in `libc/src/` without -preprocessor guards for `LLVM_LIBC_FULL_BUILD`. +preprocessor guards for `LIBC_FULL_BUILD`. Consider the case where an implementation in `libc/src/` may wish to refer to a `sigset_t`, what header should be included? ``, ``, @@ -258,16 +258,16 @@ a `sigset_t`, what header should be included? ``, ``, None of the above. Instead, code under `src/` should `#include "hdr/types/sigset_t.h"` which contains preprocessor guards on -`LLVM_LIBC_FULL_BUILD` to either include the public type (fullbuild mode) or +`LIBC_FULL_BUILD` to either include the public type (fullbuild mode) or the underlying system header (overlay mode). Implementations in `libc/src/` should NOT be `#include`'ing using `<>` or `"include/*`, except for these "proxy" headers that first check for -`LLVM_LIBC_FULL_BUILD`. +`LIBC_FULL_BUILD`. These "proxy" headers are similarly used when referring to preprocessor defines. Code under `libc/src/` should `#include` a proxy header from -`hdr/`, which contains a guard on `LLVM_LIBC_FULL_BUILD` to either include +`hdr/`, which contains a guard on `LIBC_FULL_BUILD` to either include our header from `libc/include/` (fullbuild) or the corresponding underlying system header (overlay). diff --git a/libc/docs/dev/entrypoints.md b/libc/docs/dev/entrypoints.md index 09cc1181b2a57..c0c3bbe3e5882 100644 --- a/libc/docs/dev/entrypoints.md +++ b/libc/docs/dev/entrypoints.md @@ -87,8 +87,11 @@ add_entrypoint_object( For redirecting entrypoints (e.g., when one function is a simple alias for another), the `REDIRECTED` option can be specified to the rule. -Entrypoint objects are generally forbidden from depending on other entrypoint -objects. The single allowed exception is `libc.src.errno.errno`: entrypoints +Entrypoint objects are forbidden from depending on other entrypoint objects; +see {ref}`code_organization` for what to do instead. The allowed exceptions +are listed in `check_entrypoint_deps` in +`libc/cmake/modules/LLVMLibCObjectRules.cmake`. The one that matters in +practice is `libc.src.errno.errno`: entrypoints that read or set `errno` should declare both `libc.src.__support.libc_errno` (the internal header) and `libc.src.errno.errno` (the compiled runtime storage) in their `DEPENDS` list. See {ref}`setting_errno` for detailed rules. @@ -112,5 +115,13 @@ determined by `entrypoints.txt` files located in the `libc/config` tree. If you are implementing a new entrypoint, you must add its target name to the relevant `entrypoints.txt` files for it to be included in the library build. -For more details on platform configuration, see the {ref}`porting` guide. +See {ref}`code_organization` for which files those are, and the {ref}`porting` +guide for more details on platform configuration. + +### Experimental entrypoints + +Entrypoints whose implementations are known to be incomplete are listed inside +an `if(LLVM_LIBC_ENABLE_EXPERIMENTAL_ENTRYPOINTS)` block in `entrypoints.txt`. +The option is off by default. See {ref}`extensions_and_stubs` for when an +entrypoint belongs there and when it can leave. diff --git a/libc/docs/dev/implementation_guidelines.md b/libc/docs/dev/implementation_guidelines.md new file mode 100644 index 0000000000000..7b1bd5e4f61be --- /dev/null +++ b/libc/docs/dev/implementation_guidelines.md @@ -0,0 +1,195 @@ +(implementation_guidelines)= + +# Implementation Guidelines + +{ref}`code_style` covers naming, macros, and formatting. This page covers the +judgment calls that come up while writing an implementation: what to optimize +for, how to use memory, when to check inputs, and which parts of the tree are +known to need rework. + +## Priorities + +When goals conflict, prefer them in this order: + +1. **Correctness.** Follow the standard. See {ref}`undefined_behavior` for + what to do where the standard is silent. +2. **Readability.** Code that is long but obvious is better than code that is + short and clever. The character classification tables in + `src/__support/ctype_utils.h` are long, and anyone can see what they do. +3. **Efficiency.** This encompasses latency, throughput, binary size, and memory + usage. There's no consistent formula for tradeoffs between these, but in + general baremetal users prioritize binary size and servers prioritize latency + and throughput. + +The memory functions and the math library are exceptions, where performance is +an explicit goal alongside correctness. + +Microbenchmarks are useful for finding a direction but are not a goal in and of +themselves. The ideal performance measurement is from sampling real systems, +though that's impractical for most changes. The project does not have a settled +benchmarking setup today. + +## Memory + +- **Allocate on the heap only when necessary.** Many targets have a small + heap or none. When allocation is needed, follow the allocation rules in + {ref}`code_style` and handle failure. +- **Stack buffers may be large when the function needs them.** Prefer a + correct implementation with a generous buffer over a complicated one. Sizes + can be tuned later. +- **Do not use unbounded recursion.** Baremetal users must be able to + determine the maximum stack depth statically. Recursion with an explicit + depth limit is acceptable. + +### Allocators + +LLVM-libc does not aim to provide its own general-purpose allocator for hosted +systems. On Linux the supported allocator is +[Scudo](https://llvm.org/docs/ScudoHardenedAllocator.html), and users with +specific performance requirements are expected to bring the allocator they +already prefer. On baremetal targets LLVM-libc provides an optimized allocator +in `src/__support/freelist_heap.h`. + +## Runtime checks + +- **Null pointers.** At the start of an entrypoint, pass each pointer argument + that would be dereferenced, and for which a null pointer is not a valid + input, to `LIBC_CRASH_ON_NULLPTR` (`src/__support/macros/null_check.h`). The + check is lightweight, is controlled by the `LIBC_ADD_NULL_CHECKS` option, and + exists to make a crash easy to diagnose. Put it in the entrypoint only, not + in internal code. +- **Internal invariants.** Use `LIBC_ASSERT`. It compiles away in release + builds. +- **Hardening.** `LIBC_COPT_HARDENING_MODE` was added for the allocator's + integrity checks. Extending it to other code needs discussion first. +- **Everything else.** Do not validate inputs beyond what the standard + requires. {ref}`undefined_behavior` explains the reasoning. + +## Headers + +- **Include what you use.** A `.cpp` file that names `mode_t` includes + `hdr/types/mode_t.h` itself and does not rely on another header to provide + it. +- **Use proxy headers** from `libc/hdr/` for every type and macro that the + libc also provides publicly. See the header inclusion policy in + {ref}`code_style`. Compiler-provided headers are tolerated but should also + go through a proxy header where one exists. The existing `#include + ` style includes under `libc/src/` are incorrect and should not + be copied. +- **Type traits** come from the umbrella header + `src/__support/CPP/type_traits.h`, not from the individual files under + `src/__support/CPP/type_traits/`. + +## File headers and comments + +Every file touched by a change uses the current LLVM file header: a bare +separator line, the license block, and a `\file` description. + +```c++ +//===----------------------------------------------------------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +/// +/// \file +/// Implementation of fflush. +/// +//===----------------------------------------------------------------------===// +``` + +Comment only what the code does not already say. A straightforward function +or test needs no introductory paragraph, and a function that is moved keeps +exactly the comments it had. Also, don't number the steps of a function in +its comments (`// 1. ...`, `// 2. ...`). It makes later refactors noisier than +necessary for no tangible benefit. + +## Character data + +Do not compare against character literals (except for equality), or do +arithmetic on them to classify or convert characters. Use the helpers in +`src/__support/ctype_utils.h` and `src/__support/wctype_utils.h`, which work +for non-ASCII encodings and for both `char` and `wchar_t`. See +{ref}`design_philosophy` for the model behind this. + +(math_function_guidelines)= + +## Math functions + +The {ref}`math page ` describes the goals of the math library, and +`libc/src/math/docs/add_math_function.md` is the step-by-step guide for adding +a function. The points below are the ones that most often need stating. + +### Accuracy and error reporting + +- The default implementation of every math function is expected to be + correctly rounded in all rounding modes, and to support `errno`, floating + point exceptions, and signaling NaNs. A signaling NaN input produces a quiet + NaN and raises `FE_INVALID`. +- The {doc}`math.h status tables <../headers/math>` record the accuracy of + each function. The double precision `atan` and `atan2` entries currently + list 1 ULP. That is a known gap, and it is being fixed. +- `errno` and floating point exceptions are reported only through + `fputil::set_errno_if_required` and `fputil::raise_except_if_required`. +- The GPU-specific math implementations have different accuracy bounds and + are maintained by the GPU maintainers. + +### Structure + +- The implementation lives in a header under `src/__support/math/`, and the + entrypoint is a thin wrapper. +- A function can have several implementations, each in its own namespace + (`double_eval`, `float_eval`, `float_eval_cr_rn`, `integer_eval`), with + `LIBC_MATH__IMPL` selecting the one to use. This is the intended + structure for every function, although only some functions have been + converted so far. Variants other than the default may make different + guarantees. Once more variants exist, the {ref}`math page ` will have + a table of what each one guarantees. +- The functions in `math::static_rounding` take the rounding mode as an + argument. They exist for callers that need a fixed rounding mode, such as + compilers. They are not the primary implementation of a function. The + primary implementation is whichever one is best for the target's + configuration. +- A pull request may add a function for a single floating point type. It does + not need to cover every type at once. + +### Testing + +- The author of a math change runs the unit, smoke, and exhaustive tests for + it on their own machine, at least in the default configuration, and + preferably before sending the pull request. Reviewers check that this was + done. CI covers most of the tests and some configurations, not all of them. +- The exhaustive tests take a long time and are not run by `check-libc`. +- If the tests are hard to run, or hard to get passing, ask the math + maintainers for help. +- Tests that compare against MPFR run only in overlay mode. Every function + also needs a smoke test, which is what runs in full builds and on baremetal + and GPU targets. +- `libc/test/src/math/exhaustive/` contains hard-to-round inputs from the + [CORE-MATH](https://core-math.gitlabpages.inria.fr/) project, as text files + (`.wc`) and in a compact binary form (`.bin`). The two forms must stay + consistent. A CI check for that, and instructions for adding more + hard-to-round cases, are planned. + +## Areas that need rework + +The following parts of the tree are known to fall short of the guidelines on +this page. Do not use them as models for new code. Improvements are welcome +as separate, focused changes. + +- **The `File` abstraction.** The Linux implementation is in + `src/__support/File/`, while the baremetal and GPU implementations are in + `src/stdio/baremetal/` and `src/stdio/gpu/`. These should share one design. +- **`src/__support/HashTable/`** needs a cleanup of its implementation style, + its header inclusion, and its architecture-specific dispatch. +- **String to floating point conversion.** The layering between the parsers + and their fallbacks should be clearer. Today the original string has to be + passed along even after the first parser has consumed it. +- **`src/__support/StringUtil/`** holds only the `strerror` and `strsignal` + text and should be renamed for what it contains. +- **Separation by OS dependence.** See {ref}`code_organization`. +- **`libc/benchmarks/`** is unmaintained, apart from the GPU benchmarks in + `libc/benchmarks/gpu/`, and will be removed or rewritten. +- **`libc/fuzzing/`** is in use but covers less than it should. diff --git a/libc/docs/dev/implementation_standard.md b/libc/docs/dev/implementation_standard.md index d4bdca3c91b41..b5aa1050ad942 100644 --- a/libc/docs/dev/implementation_standard.md +++ b/libc/docs/dev/implementation_standard.md @@ -62,7 +62,9 @@ LLVM_LIBC_FUNCTION(int, isalpha, (int c)) { Notice the use of the macro `LLVM_LIBC_FUNCTION`. This macro helps us define a C alias symbol for the C++ implementation. For example, for a library build, -the macro is defined as follows: +the macro expands to roughly the following. The full definition, including the +optional fourth argument that names the C alias, is in +`src/__support/common.h`. ``` #define LLVM_LIBC_FUNCTION(type, name, arglist) diff --git a/libc/docs/dev/implementing_a_function.md b/libc/docs/dev/implementing_a_function.md index 26e5468cde028..d6dd477096d95 100644 --- a/libc/docs/dev/implementing_a_function.md +++ b/libc/docs/dev/implementing_a_function.md @@ -8,6 +8,10 @@ This guide provides a step-by-step walkthrough for adding a new function to LLVM Adding a new function involves several steps, from updating the public specification to implementing and testing the code. Below is the standard checklist for contributors. +Before starting, read {ref}`code_organization` and {ref}`patterns_to_avoid`. +The most common mistake in a new function is calling another public function +from it, which is not allowed. + ## Step-by-Step Checklist ### 1. Header Entry @@ -33,7 +37,10 @@ Write the actual code for the function. - **File**: `libc/src/
/.cpp` (or `libc/src/
//.cpp` for platform-specific implementations) - Use the `LLVM_LIBC_FUNCTION` macro. -- Refer to {ref}`code_style` for naming and layout conventions. +- Refer to {ref}`code_style` for naming and layout conventions and to + {ref}`implementation_guidelines` for memory use and input checking. +- A function that makes a system call does so through a wrapper. See + {ref}`syscall_wrapper_refactor`. ### 4. CMake Rule @@ -50,7 +57,9 @@ Add a CMake target for the new function so it can be compiled. Register the new entrypoint for the target platforms to include it in the build. - **File**: `libc/config///entrypoints.txt` -- Add the new function to the list of active entrypoints. +- Add the new function to the list of active entrypoints, on every target + where it works. See {ref}`code_organization` for the minimum set and for + the difference between the overlay and full build lists. ### 6. Testing @@ -61,3 +70,5 @@ Create tests to verify the implementation. setting `errno`, use `ErrnoCheckingTest` and `ErrnoSetterMatcher` (see {ref}`setting_errno`). - Update the `CMakeLists.txt` in the test directory (`libc/test/src/
/CMakeLists.txt`) to include the new test target. +- See {ref}`testing_guide` for what the test should cover and how to confirm + that it ran. diff --git a/libc/docs/dev/index.md b/libc/docs/dev/index.md index fc9711c2ab5cf..8c6879c1985c6 100644 --- a/libc/docs/dev/index.md +++ b/libc/docs/dev/index.md @@ -9,6 +9,10 @@ Navigate to the links below for information on the respective topics: building_docs code_style +code_organization +implementation_guidelines +patterns_to_avoid +testing_guide source_tree_layout entrypoints implementing_a_function diff --git a/libc/docs/dev/patterns_to_avoid.md b/libc/docs/dev/patterns_to_avoid.md new file mode 100644 index 0000000000000..5d1cd4f265329 --- /dev/null +++ b/libc/docs/dev/patterns_to_avoid.md @@ -0,0 +1,61 @@ +(patterns_to_avoid)= + +# Common Mistakes and Legacy Patterns + +LLVM-libc's conventions have changed over the years, and the tree still +contains code written to older ones. **Existing code is not precedent.** A +pattern being present in the tree, even many times, does not make it +acceptable in a new change. This page lists the mistakes reviewers see most +often and the old patterns that should not be copied. + +This applies with particular force to changes written with AI assistance, +because a tool that imitates nearby code will imitate its outdated parts too. + +## Most common mistakes + +1. **Calling one entrypoint from another.** It is the natural way to write + many functions and it is not allowed. See {ref}`code_organization`. +2. **Mimicking another libc's interface.** Changing a type's layout, a + function's observable behavior, or the set of exported symbols solely to + match another libc is sometimes the quickest way to get an existing program + running. That matching is not the intent for LLVM-libc, and goes against its + design principles. See: {ref}`making_existing_software_work`. +3. **Copying an outdated pattern from old code.** See the table below. +4. **Copying from another libc.** This is not allowed for licensing reasons, + and translated C does not fit this code base. See + {ref}`design_philosophy`. +5. **Leaving out part of the registration.** A new function needs an entry in + `libc/include/
.yaml` and in the relevant `entrypoints.txt` files in + addition to its implementation. See {ref}`implementing_a_function`. + +## Legacy patterns + +| Seen in the tree | Write this instead | +| --- | --- | +| A first line of the form `//===-- Title ---...---===//` in a file header | The current LLVM file header, shown in {ref}`implementation_guidelines`. Update the header of any file you touch. | +| `syscall_impl(...)` in an entrypoint, followed by a hand-written check of the result | A wrapper in `src/__support/OSUtil//syscall_wrappers/` that uses `syscall_checked`. See {ref}`syscall_wrapper_refactor`. | +| Functions and variables inside an anonymous namespace | `static` functions and variables. Anonymous namespaces are for types only, as in the [LLVM coding standards](https://llvm.org/docs/CodingStandards.html#restrict-visibility). | +| A `.cpp` file that gets its types from the implementation header it includes | Include what you use: each file includes the proxy header for every type and macro it names. | +| `#include ` or another public header in `libc/src/` | The proxy headers in `libc/hdr/`. | +| `libc_errno` assigned inside an internal helper | The helper returns `ErrorOr` and the entrypoint assigns `libc_errno`. | +| A new file or directory named `utils` | A name that says what the code does. | +| `ASSERT_TRUE(a == b)` and `ASSERT_TRUE(p != nullptr)` in tests | `ASSERT_EQ(a, b)` and `ASSERT_NE(p, nullptr)`, which print both values on failure. | +| A bare path literal such as `"testdata/file.txt"` or `"/nonexistent"` in tests | `libc_make_test_file_path(...)`, including for paths that are meant not to exist. | + +## Code that is hard to review + +These are not style violations in the formatting sense. They make code hard to +reason about, and reviewers will ask for them to be rewritten. + +- **Deeply nested control flow.** An `if` inside a loop inside a loop inside + an `if` is difficult to follow. Collapse conditions, return early, and move + inner blocks into helper functions. +- **`while (true)` with a single `break`.** The condition that triggers the + `break` is the loop condition. Write it there. +- **Unnecessary lambdas.** Use a lambda when a callable is needed. For + ordinary code reuse, write a function. +- **Templates and variadic forwarding where a fixed signature would do.** +- **Magic constants.** A bare `7`, `61`, or `0xFFF` in an expression leaves + the reader to work out what it stands for. Give every such constant a name + that says what it means. Multiplying by `DAYS_PER_WEEK` explains itself; + multiplying by `7` does not. diff --git a/libc/docs/dev/source_tree_layout.md b/libc/docs/dev/source_tree_layout.md index 45ce04d3b8cfd..7a4b8940c1554 100644 --- a/libc/docs/dev/source_tree_layout.md +++ b/libc/docs/dev/source_tree_layout.md @@ -16,6 +16,7 @@ directories: - hdr - include - lib + - shared - src - startup - test @@ -67,8 +68,9 @@ in fullbuild or overlay mode. The `include` directory contains: -1. `*.h.def` files - These files are used to construct the generated public - header files. +1. `*.yaml` files - These describe the contents of each public header. The + public headers are generated from them; see {ref}`header_generation`. A + few headers also have a `*.h.def` template file. 2. Self contained public header files - These are header files which are already in the form that get installed when LLVM-libc is installed on a user's computer. These are mostly in the `llvm-libc-macros` and @@ -79,6 +81,12 @@ The `include` directory contains: This directory contains a `CMakeLists.txt` file listing the targets for the public libraries `libc.a`, `libm.a` etc. +## The `shared` directory + +This directory contains the headers through which other LLVM projects use +LLVM-libc's internal code. See {ref}`hand_in_hand`, and +{ref}`code_organization` for the constraints on code exposed this way. + ## The `src` directory This directory contains the implementations of the llvm-libc entrypoints. For @@ -92,6 +100,8 @@ more details on what an entrypoint is and how it is implemented, see the corresponding to the public `math.h` header file is named `math`. The implementation standard document explains more about the *header* directories. +3. The `__support` directory contains internal code shared across the + library. {ref}`code_organization` explains what belongs there. ## The `startup` directory diff --git a/libc/docs/dev/syscall_wrapper_refactor.md b/libc/docs/dev/syscall_wrapper_refactor.md index 7c84915eba62f..5d63f4875369b 100644 --- a/libc/docs/dev/syscall_wrapper_refactor.md +++ b/libc/docs/dev/syscall_wrapper_refactor.md @@ -1,14 +1,15 @@ (syscall_wrapper_refactor)= -# Syscall Wrapper Refactoring +# Syscall Wrappers ## Purpose -LLVM-libc is transitioning to a centralized system for Linux syscalls. The goal -is to move all direct `syscall_impl` calls into a dedicated directory: -`src/__support/OSUtil/linux/syscall_wrappers/`. +All Linux system calls in LLVM-libc are made through wrapper functions in one +directory: `src/__support/OSUtil/linux/syscall_wrappers/`. Code outside +`src/__support/OSUtil/` does not call `syscall_impl` or use `SYS_*` numbers +directly. -This refactor provides several benefits: +This provides several benefits: - **Reusability**: Allows multiple entrypoints to share a single syscall implementation without public APIs depending on other public APIs. @@ -19,12 +20,15 @@ This refactor provides several benefits: - **Maintainability**: Centralizes platform-specific syscall logic, making it easier to audit and update. +The migration of existing code to this structure is still in progress, so the +tree contains entrypoints that call `syscall_impl` themselves. Do not copy +them. New code uses a wrapper, and adds one if it does not exist yet. + ## The Pattern -Each syscall should have its own header-only library in the `syscall_wrappers` -directory. The wrapper function should return an `ErrorOr`. Wrappers live -in the `linux_syscalls` namespace to make call sites self-documenting and to -clearly identify any leakage into OS-generic code. +Each syscall has its own header-only library in the `syscall_wrappers` +directory. Wrappers live in the `linux_syscalls` namespace to make call sites +self-documenting and to clearly identify any leakage into OS-generic code. ### Example Wrapper (`src/__support/OSUtil/linux/syscall_wrappers/read.h`): @@ -47,22 +51,53 @@ LIBC_INLINE ErrorOr read(int fd, void *buf, size_t count) { } // namespace LIBC_NAMESPACE_DECL ``` -## How to Migrate +The entrypoint turns the result into the public contract. In outline: -1. **Cleanup Existing Implementation**: If the syscall was previously - implemented in `OSUtil/linux/fcntl.cpp` (or similar), remove the old - implementation to replace it with the new wrapper. +```c++ +LLVM_LIBC_FUNCTION(ssize_t, read, (int fd, void *buf, size_t count)) { + ErrorOr result = linux_syscalls::read(fd, buf, count); + if (!result) { + libc_errno = result.error(); + return -1; + } + return *result; +} +``` -2. **Create the Wrapper**: Add a new header file in +### Rules for wrappers + +- **A wrapper represents an operation of the operating system**, not a POSIX + function. Linux-specific system calls get wrappers like any other. +- **A wrapper hides which syscall number implements the operation.** Some + architectures lack the older syscalls (there is no `SYS_mkdir` on AArch64, + for instance), so the `mkdir` wrapper calls `SYS_mkdirat`. Callers do not + repeat that choice. +- **Use `syscall_checked`.** It converts the kernel's error convention into an + `ErrorOr`. Do not call `syscall_impl` and check the sign of the result by + hand. +- **Syscalls that cannot fail return the plain type.** `getpid` returns + `pid_t`, not `ErrorOr`, and is one of the few places that use + `syscall_impl`. +- **A wrapper never sets `errno`.** The entrypoint does. See + {ref}`setting_errno`. + +Other operating systems follow the same structure under their own directory +and namespace, for example `src/__support/OSUtil/freebsd/syscall_wrappers/`. + +## How to Migrate + +1. **Create the Wrapper**: Add a new header file in `src/__support/OSUtil/linux/syscall_wrappers/`. -3. **Update CMake**: Add a `add_header_library` target for the new wrapper in +2. **Update CMake**: Add a `add_header_library` target for the new wrapper in `src/__support/OSUtil/linux/syscall_wrappers/CMakeLists.txt`. -4. **Refactor Entrypoints**: +3. **Refactor Entrypoints**: - Include the new wrapper header (e.g., `read.h`). - Replace direct `syscall_impl` calls with `linux_syscalls::`. - Update the entrypoint's `DEPENDS` in `CMakeLists.txt` to include the new wrapper target. + +Migrate one system call per pull request. diff --git a/libc/docs/dev/testing_guide.md b/libc/docs/dev/testing_guide.md new file mode 100644 index 0000000000000..5910546f6b0d7 --- /dev/null +++ b/libc/docs/dev/testing_guide.md @@ -0,0 +1,160 @@ +(testing_guide)= + +# Testing Guide + +This page describes how LLVM-libc tests are organized, what a test for a new +function should cover, and how to make sure the test you wrote is the test +that ran. {ref}`build_and_test` lists the build commands. + +## Kinds of tests + +All tests are declared with `add_libc_test` or one of its wrappers, in the +`CMakeLists.txt` next to the test source under `libc/test/`. + +| Rule | Use it for | +| --- | --- | +| `add_libc_test` | The default. The same source is built against the system libc in overlay mode and against LLVM-libc's own startup code and headers in a full build. | +| `add_fp_unittest` | Floating point tests. Adds the floating point test helpers. | +| `add_integration_test` | Behavior that needs a real process from start to finish: program startup, `exec`, thread-local storage, threads still running at exit. Full build only. | + +A test may be restricted with `OVERLAY_ONLY` or `FULL_BUILD_ONLY`. Use these +only when the test cannot work in the other mode. An entrypoint that is +full-build only does not need the option: its test is skipped automatically in +overlay mode because the entrypoint is missing. + +Each test names the `SUITE` it belongs to, and each test directory defines a +suite named after it, such as `libc-ctype-tests`. + +## Running tests + +```sh +# Build and run one test. +ninja -C build libc.test.src.ctype.isalpha_test + +# Build and run every test in one directory. +ninja -C build libc-ctype-tests + +# Build and run everything. +ninja -C build check-libc +``` + +While developing, run the specific test. Before considering a change finished, +run `check-libc`, because a change in one area sometimes affects another. + +A change should pass in both an overlay build and a full build. Keeping one +build directory for each mode makes this quick. + +## Making sure your test ran + +CMake silently drops a test when one of its dependencies is not available on +the target: an entrypoint that is not in `entrypoints.txt`, a missing startup +object, or a missing MPFR library. A dropped test does not fail. It is simply +not part of `check-libc`, which then passes. This is expected behavior, since +some entrypoints are unavailable on some targets and should not report an error. + +To avoid reporting a pass for a test that never ran: + +- Configure with `-DLIBC_CMAKE_VERBOSE_LOGGING=ON`. CMake then prints a line + starting with `Skipping` for every test dropped because of a missing + entrypoint or startup object. Tests dropped because MPFR is missing are + reported only at CMake's verbose log level (`--log-level=VERBOSE`). +- Run the test by its own target name. If the target does not exist, the test + was dropped. +- Watch it fail first. Run the test before the implementation is complete, or + break the implementation on purpose, and confirm that the test fails. + +## What to test + +Test the behavior that LLVM-libc is responsible for. + +- **A function that is a thin wrapper around a system call** needs one test of + success and one of failure. That shows the arguments reach the kernel in the + right order and that an error is reported through the return value and + `errno`. Do not test path resolution, permission checks, or other behavior + the kernel implements. +- **Logic that lives in the libc** is tested thoroughly, including the boundary + cases. A search of `PATH` in `execvp` and the buffering in `stdio` are + examples. +- **Tests for C standard functions use only C standard functions** to + set up and check their state. A test for `fgets` should not need `pipe` or + `fdopen`. + +A test file is generally named after the function it tests. When one file +tests several functions that only make sense together, name it after all of +them, for example `popen_pclose_test.cpp`. This is a general rule and not an +absolute one: `fileop_test.cpp` covers many `stdio` functions under one +descriptive name. + +### Conventions + +- Test suite names start with `LlvmLibc`. +- Call the LLVM-libc functions through the `LIBC_NAMESPACE` namespace to avoid + pulling in the system libc's implementations. +- Include types and macros through the proxy headers in `libc/hdr/`, as in + implementation code. +- For functions that set `errno`, derive the fixture from `ErrnoCheckingTest` + and check results with the `Succeeds` and `Fails` matchers from + `test/UnitTest/ErrnoSetterMatcher.h`. The fixture asserts that `errno` is + zero when each test ends, so a test that expects an error must consume it. +- Compare with `ASSERT_EQ`, `ASSERT_NE`, and the other typed assertions, not + `ASSERT_TRUE(a == b)`. +- Use `cpp::scope_exit` to close files and remove temporary files when a later + assertion could return early. + +## Tests must not depend on their environment + +Tests run on developers' machines, in containers, under emulators, and on +buildbots, as different users. + +- **Paths.** Create every path with `libc_make_test_file_path`, including + paths that are meant not to exist. Never assume that a fixed absolute path + such as `/nonexistent` is absent. +- **Privileges.** Avoid tests whose result depends on the user's privileges. + Where that is unavoidable, the test must pass both as root and as an + ordinary user. +- **System state.** Do not rely on `/proc` being mounted or on network access. +- **Emulators.** A test that cannot work under user-mode emulation is guarded + with `LIBC_TEST_UNDER_EMULATOR` instead of having its expectations loosened. + +## Disabling a test + +A test that cannot pass on some target is disabled in CMake, with a condition +that names the target and a link to an issue that tracks re-enabling it. Do +not comment out a test body or wrap it in `#if 0`. + +## Death tests and null checks + +`EXPECT_DEATH` and `EXPECT_EXITS` tests, including the tests that check +`LIBC_CRASH_ON_NULLPTR`, are wanted. Some CI configurations currently skip +them with `LIBC_TEST_SKIP_DEATH_TESTS`; that is a limitation of those +configurations to be fixed and not a reason to leave such tests out. + +## Other test frameworks + +Tests are written against LLVM-libc's own framework in `libc/test/UnitTest/`, +which is a subset of GoogleTest. Some downstream users compile the same tests +with GoogleTest or a compatible framework, so stay within that subset and the +matchers LLVM-libc provides. Keeping a downstream framework working is the +responsibility of that downstream. + +## Math tests + +- Tests that compare against MPFR (`NEED_MPFR`) run in overlay mode only, + because MPFR is not yet supported in a full build. A math change must be + tested in an overlay build with MPFR installed. +- Tests in `libc/test/src/math/smoke/` do not need MPFR. They are the math + tests that run in full builds and on baremetal and GPU targets. +- Tests in `libc/test/src/math/exhaustive/` take a long time and are not run + by `check-libc`. The author of a math change runs them locally. See + {ref}`math_function_guidelines`. + +## Fuzzing and benchmarks + +Neither is required for a new function today. Ideally, security-sensitive +functions have fuzz tests and performance-sensitive functions have benchmarks. + +Fuzz tests live in `libc/fuzzing/` (see {ref}`fuzzing`). A migration of the +fuzz tests to [FuzzTest](https://github.com/google/fuzztest) is being +considered. The benchmarks under +`libc/benchmarks/` are unmaintained, apart from the GPU benchmarks in +`libc/benchmarks/gpu/`. diff --git a/libc/docs/dev/undefined_behavior.md b/libc/docs/dev/undefined_behavior.md index 9a2fb7ac8b495..8eacaef566fe8 100644 --- a/libc/docs/dev/undefined_behavior.md +++ b/libc/docs/dev/undefined_behavior.md @@ -3,11 +3,13 @@ # Defining Undefined Behavior The C standard leaves behavior undefined or implementation defined in many -places. Undefined behavior is behavior that the standards leave up to the -implementation. As an implementation, LLVM's libc must provide a result for any -input, including inputs for which the result is undefined. This page provides -examples of how these situations are handled in existing code, followed by -guidelines to help determine the right approach for new situations. +places. Undefined behavior is behavior for which the standard imposes no +requirements, and implementation-defined behavior is behavior that each +implementation chooses and documents. As an implementation, LLVM's libc must +provide a result for any input, including inputs for which the result is +undefined. This page provides examples of how these situations are handled in +existing code, followed by guidelines to help determine the right approach for +new situations. ## Guidelines @@ -174,8 +176,8 @@ For LLVM-libc, `tdelete` returns bit-casted `uintptr_t`'s maximum value. ### `twalk/twalk_r/tdestroy` with Null Function Pointer The standard requires that `twalk`, `twalk_r`, and `tdestroy` -to be used with a valid function pointer. LLVM-libc follows the behavior of -configured via the `LIBC_ADD_NULL_CHECKS` option. +to be used with a valid function pointer. LLVM-libc's behavior for a null +function pointer is configured via the `LIBC_ADD_NULL_CHECKS` option. ### Invalid Thread Joining Behavior diff --git a/libc/docs/full_cross_build.md b/libc/docs/full_cross_build.md index 4783b203184ea..e4331afd857de 100644 --- a/libc/docs/full_cross_build.md +++ b/libc/docs/full_cross_build.md @@ -161,6 +161,15 @@ component of the target triple as `none`. For example, to build for a `arm-none-eabi`. Other than that, the libc for a bare metal target can be built using any of the three recipes described above. +A baremetal toolchain that ships several variants of the library (multilibs) +may not want a complete copy of the libc in each one. Setting +`-DLLVM_LIBC_ENABLE_MINOR_VARIANT=ON` builds, instead of the usual archives, a +single archive that contains only the entrypoints listed in +`LLVM_LIBC_SUBARCHIVE_ENTRYPOINTS`. The archive is named `libllvm.a`, +where `` is the value of `LLVM_LIBC_SUBARCHIVE_NAME`. Every listed +entrypoint must be enabled in the target's `entrypoints.txt`, and the startup +objects are not installed for such a build. + ## Building for the GPU To build for a GPU architecture, it should only be necessary to specify the diff --git a/libc/docs/hand_in_hand.md b/libc/docs/hand_in_hand.md index 7e91b90d5eef1..fd2b4a4cf6a0c 100644 --- a/libc/docs/hand_in_hand.md +++ b/libc/docs/hand_in_hand.md @@ -34,7 +34,14 @@ Current Hand-in-Hand users: - Libc++ uses it for `from_chars`. - OpenMP uses it for printf on GPUs. -- [WIP] clang uses it for APFloat functions. +- compiler-rt can use it to implement its floating point builtins + (`COMPILER_RT_USE_LIBC_MATH`). +- The offloading runtime and the sanitizers' GPU support use the RPC headers. +- [WIP] LLVM uses it for APFloat functions. A few functions are provided so + far. + +See {ref}`code_organization` for the constraints on code that is exposed this +way. For more information check out the 2024 talk about the original Project: diff --git a/libc/docs/index.md b/libc/docs/index.md index 545f0b2974027..2be339723c46f 100644 --- a/libc/docs/index.md +++ b/libc/docs/index.md @@ -100,6 +100,8 @@ compiler_support :maxdepth: 1 contributing +design_philosophy +abi_policy build_and_test dev/index math/index diff --git a/libc/docs/platform_support.md b/libc/docs/platform_support.md index 87f98d3a649d0..56525589248e2 100644 --- a/libc/docs/platform_support.md +++ b/libc/docs/platform_support.md @@ -1,11 +1,54 @@ +(platform_support)= + # Platform Support -Development is currently mostly focused on Linux. MacOS and Windows has -partial support, but has bitrot and isn't being tested continuously. +LLVM-libc's targets differ in how complete they are. This page describes the +status of each. {doc}`arch_support` lists the +architectures that are tested continuously, and {ref}`porting` describes what +a target needs in order to be added or kept. + +## Used in production + +These targets have, or are expected to soon have, users deploying LLVM-libc in +a production environment. They are given some amount of priority over the more +experimental targets. + +| Operating system | Architectures | +| --- | --- | +| Linux | x86-64, AArch64, RISC-V | +| Baremetal | AArch64, RISC-V, 32-bit Arm | + +New functionality should be enabled on all platforms when possible, but may be +rolled out to just these targets first as the primary development targets. See +{ref}`code_organization`. + +## Other targets -LLVM-libc is currently being integrated into Android and Fuchsia operating +The remaining targets are partially complete or experimental. Breakage on them +is still a bug, but contributors are not expected to be able to test them. + +| Target | Status | +| --- | --- | +| GPU (AMDGPU, NVPTX) | Maintained by the GPU maintainers. See {doc}`the GPU docs `. | +| Baremetal Hexagon | Newer target (as of 2026) but hopefully an area of growth. | +| FreeBSD | A port is in progress. It has a maintainer and presubmit CI. | +| macOS and Windows | Overlay mode only. The overlay tests run in presubmit, but both ports need significant work, and a full build for these systems has not been designed. | +| Linux on 32-bit Arm | Primarily used for testing on 32-bit Arm since baremetal CI is challenging. | +| Linux on PowerPC | Minimal. It exists mainly so that the code shared with libc++ keeps building there. | +| Linux on i386 | Theoretically [in the works](https://github.com/llvm/llvm-project/issues/93709), though it may be sunset. | +| UEFI | Experimental and incomplete. See {doc}`the UEFI docs `. | + +LLVM-libc is also being integrated into the Android and Fuchsia operating systems via {doc}`overlay mode `. +## Breakages + +A change that breaks any buildbot should be reverted or, if the fix is trivial, +fixed forward. This is intended to follow LLVM's +[patch reversion policy](https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy). + +## Operating system versions + For Linux, we support kernel versions as listed on [kernel.org](https://kernel.org/), including `longterm` (not past EOL date), `stable`, and `mainline` versions. We actively adopt new features @@ -17,5 +60,3 @@ For Windows, we plan to support products within their lifecycle. Please refer to LLVM-libc does not guarantee backward compatibility with operating systems that have reached their EOL. Compatibility patches for obsolete operating systems will not be accepted. - -For GPU, reference {doc}`our GPU docs `. diff --git a/libc/docs/porting.md b/libc/docs/porting.md index 96f122bbb61d0..ea100f6abd299 100644 --- a/libc/docs/porting.md +++ b/libc/docs/porting.md @@ -74,6 +74,35 @@ config and the [x86_64](https://github.com/llvm/llvm-project/tree/main/libc/config/linux/x86_64) config. +(custom_config_directory)= + +### Using a config directory outside the source tree + +The configuration for a build does not have to live in `libc/config`. The +`LIBC_CONFIG_PATH` CMake option names a directory to use instead of the +in-tree directory for the target: + +```sh +cmake ... -DLIBC_CONFIG_PATH=/path/to/my/config +``` + +The directory must contain an `entrypoints.txt`, and a `headers.txt` for a +full build. An `exclude.txt` and a `config.json` are used if they are +present. The `config.json` is applied on top of the defaults in +`libc/config/config.json`, and the in-tree `config.json` files for the target +are not read. + +This is the way to build a libc with a different set of functions, headers, +and options than the default for a target without editing the in-tree +configuration. It is not widely used yet. + +### Building without `-Werror` + +LLVM-libc is compiled with `-Werror`. While bringing up a new target or +compiler, before the build is free of warnings, pass `-DLIBC_WNO_ERROR=ON` to +keep the warnings without treating them as errors. This should not be considered +a long term solution, the warnings should be fixed before landing upstream. + ## Upstreaming Adding a target to the main LLVM-libc has some requirements to ensure that the @@ -88,7 +117,7 @@ people are the maintainers, and they are responsible for keeping their target in good shape. This means fixing their target when it breaks, reviewing patches related to their target, and keeping the target's CI running. -Maintainers are listed in libc/maintainers.md and must follow +Maintainers are listed in `libc/Maintainers.md` and must follow [LLVM's maintainer policy](https://llvm.org/docs/DeveloperPolicy.html#maintainers). ### CI builders