From 0b56aabfb45b900c594a4ada97b2f3d32c16c9aa Mon Sep 17 00:00:00 2001 From: Scofield626 Date: Tue, 16 Jun 2026 15:21:49 +0000 Subject: [PATCH 1/4] docs: thread pool --- book/src/SUMMARY.md | 1 + book/src/advanced-tasks.md | 6 +- book/src/performance-basics.md | 17 +- book/src/task-graph.md | 5 + book/src/thread-pools.md | 218 ++++++++++++++++++++++++ book/src/troubleshooting-performance.md | 10 +- 6 files changed, 245 insertions(+), 12 deletions(-) create mode 100644 book/src/thread-pools.md diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index 081f9e4..698d68f 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -58,3 +58,4 @@ - [Performance Basics](./performance-basics.md) - [Reading Performance Metrics](./reading-performance-metrics.md) - [Troubleshooting Performance](./troubleshooting-performance.md) +- [Thread Pools, Affinity, and Real-Time Scheduling](./thread-pools.md) diff --git a/book/src/advanced-tasks.md b/book/src/advanced-tasks.md index ce0283f..5811571 100644 --- a/book/src/advanced-tasks.md +++ b/book/src/advanced-tasks.md @@ -109,7 +109,11 @@ internal state. The `type Resources<'r>` associated type declares what **hardware or system resources** a task needs. Resources represent physical endpoints -- serial ports, GPIO controllers, SPI -buses, cameras -- or shared services like a thread pool. +buses, cameras -- or shared services bound through a resource bundle. + +> Worker thread pools used to be declared as a resource. They now live in the `runtime` +> section of the config -- see +> [Thread Pools, Affinity, and Real-Time Scheduling](./thread-pools.md). ### No resources diff --git a/book/src/performance-basics.md b/book/src/performance-basics.md index 2e6ec71..52a8b14 100644 --- a/book/src/performance-basics.md +++ b/book/src/performance-basics.md @@ -109,7 +109,7 @@ This is the fast path for readers who already know what kind of bottleneck they | `async-cl-io` | Moves CopperList serialization and logging to a dedicated thread | Task latencies are fine, but end-of-CL overhead is too high | | `parallel-rt` | Pipelines multiple CopperLists across generated process stages | Several CPU-bound stages are back to back and the global rate is too low | | `mmap-fsync` | Forces file `sync_all()` on section flush | The system runs fine for a while, then collapses under dirty-page / writeback pressure | -| `background: true` | Runs one compatible source or task on the background threadpool and returns `None` while it is still busy | One isolated stage is too slow, but it does not need to block every cycle | +| `background: true` | Runs one compatible source or task on the `background` thread pool and returns `None` while it is still busy | One isolated stage is too slow, but it does not need to block every cycle | | Task-local thread pool / parallel `for` | Parallelizes the internals of one task | One hot task has obvious internal data parallelism | | `logging: (enabled: false)` or `enable_task_logging: false` | Reduces logged bytes | Bandwidth and serialized size are too high | | `logging.copperlist_count` | Increases the number of preallocated in-flight CopperLists | Async or parallel modes stall because they run out of CopperList slots | @@ -142,8 +142,8 @@ cargo run --features mmap-fsync same time, and each generated process stage keeps FIFO order for determinism. `background: true` is a **stage-level escape hatch**. One source or task moves to the -background threadpool and may return `None` for some cycles while its previous run is still -finishing. +`background` thread pool and may return `None` for some cycles while its previous run is +still finishing. ### `parallel-rt` is not the same as task-local parallelism @@ -154,10 +154,15 @@ Task-local parallelism parallelizes the **inside of one task**. If one task is much heavier than the rest, fixing that task directly is often better than adding runtime-level parallelism around it. -## Background tasks and the threadpool bundle +## Background tasks and thread pools -If a source or task is marked with `background: true`, Copper ensures a `threadpool` resource bundle is -available. If you do not provide one explicitly, the runtime creates a default one. +If a source or task is marked with `background: true`, it runs on a worker thread from the +`background` thread pool. If you do not declare that pool, the runtime creates a default one +for you. This makes background stages easy to turn on, but it does not mean they are always the right choice. They work best when skipping or sampling intermediate cycles is acceptable. + +You can size the `background` pool, pin it to specific cores, or run a stage on a separate +pool entirely. See [Thread Pools, Affinity, and Real-Time Scheduling](./thread-pools.md) for +the `runtime.thread_pools` configuration. diff --git a/book/src/task-graph.md b/book/src/task-graph.md index 43cff7a..2ecae29 100644 --- a/book/src/task-graph.md +++ b/book/src/task-graph.md @@ -115,6 +115,11 @@ Beyond `id` and `type`, each task entry supports several optional fields: ), ``` + To run a task on a specific thread pool instead of the default `background` one, use the + object form `background: (pool: "name")`. See + [Thread Pools, Affinity, and Real-Time Scheduling](./thread-pools.md) for declaring pools + with custom sizing, CPU affinity, and real-time priority. + - **`logging`** -- Controls whether Copper's unified logger records the output messages of this task. Set `enabled: false` to reduce log size for high-frequency or uninteresting tasks. diff --git a/book/src/thread-pools.md b/book/src/thread-pools.md new file mode 100644 index 0000000..2b35b86 --- /dev/null +++ b/book/src/thread-pools.md @@ -0,0 +1,218 @@ +# Thread Pools, Affinity, and Real-Time Scheduling + +By default, Copper runs your task graph on one thread: the global loop executes +source -> processing -> sink back-to-back every cycle. Two features step outside that +single thread -- `background: true` tasks and the `parallel-rt` runtime -- and both need +worker threads to run on. This chapter is about controlling those workers: how many there +are, which CPU cores they run on, and what scheduling priority the OS gives them. + +On a laptop the defaults are fine. On a real robot, where a latency-critical control loop +shares a machine with heavy background work (logging, vision, networking), you often want +to **isolate** the critical work onto dedicated cores and give it real-time priority so the +OS never lets background work preempt it. That is what thread pools, CPU affinity, and +real-time scheduling are for. + +> **Note**: This is a `std`-only, host-side feature. On `no_std` / embedded targets there +> are no threads to schedule and this whole chapter does not apply. + +## Declaring thread pools + +Thread pools are declared in the `runtime` section of `copperconfig.ron`, next to +`rate_target_hz`: + +```ron +runtime: ( + thread_pools: [ + ( id: "rt", threads: 4, affinity: [2, 3, 4, 5], policy: Fifo(priority: 80) ), + ( id: "background", threads: 2, affinity: [0, 1] ), + ( id: "vision", threads: 2, policy: Nice(10), on_error: Strict ), + ], +), +``` + +Each pool has: + +- **`id`** -- a unique name. Tasks and the runtime refer to pools by this name. Two names + are reserved (see below): `background` and `rt`. +- **`threads`** -- how many worker threads the pool runs. +- **`affinity`** (optional) -- the CPU cores the workers may run on. +- **`policy`** (optional) -- the OS scheduling policy and priority. Defaults to `Fair`. +- **`on_error`** (optional) -- what to do if the affinity or policy cannot be applied. + Defaults to `Warn`. + +> Earlier versions of Copper configured background threads through a `threadpool` +> **resource bundle**. That still works -- an existing `threadpool` resource is migrated +> automatically into a `background` pool -- but `runtime.thread_pools` is the way forward. + +## Scheduling policies + +The `policy` controls how the OS scheduler treats the pool's worker threads. On Linux these +map directly onto the POSIX scheduling policies. + +| Policy | What it is | When to use it | +|---|---|---| +| `Fair` | Normal fair time-sharing (`SCHED_OTHER` / CFS). The OS shares the CPU across threads and nothing starves. | The default. Everything that is not latency-critical. | +| `Nice(n)` | Fair scheduler with a niceness bias, `n` in `-20..=19` (lower is more favorable). A soft hint, not a guarantee. | Bias a pool below or above normal work without leaving the fair scheduler -- e.g. `Nice(10)` for heavy work that should yield to the control loop. | +| `Fifo(priority: p)` | `SCHED_FIFO` hard real-time, `p` in `1..=99`. Runs ahead of every fair thread and is **not** time-sliced -- it runs until it blocks or a higher-priority thread preempts it. | The latency-critical pipeline. Pin it with `affinity` so a busy worker cannot starve other work on the same core. | +| `RoundRobin(priority: p)` | `SCHED_RR`, same as `Fifo` except threads at the same priority are time-sliced in turn. | Several real-time workers that share a priority and should interleave fairly. | + +> **Rule of thumb**: reach for `Nice` first. Real-time policies (`Fifo` / `RoundRobin`) are +> powerful but easy to misuse -- a busy `Fifo` worker with no `affinity` can starve the rest +> of the system. Use them for genuinely time-critical work, and pin them. + +Real-time policies are **Linux-only** and usually require the `CAP_SYS_NICE` capability (run +as root, grant the capability, or raise the limit in `/etc/security/limits.conf`). On other +platforms only `Fair` and `Nice` apply; requesting `Fifo` / `RoundRobin` fails according to +the pool's `on_error` setting. + +## CPU affinity + +`affinity` is a list of logical CPU cores the pool's workers are allowed to run on. When +set, workers are pinned **Spread**: worker `i` is pinned to `affinity[i % affinity.len()]`. +So a pool with `threads: 4` and `affinity: [2, 3, 4, 5]` pins one worker to each of +cores 2-5: + +```text +worker 0 -> core 2 +worker 1 -> core 3 +worker 2 -> core 4 +worker 3 -> core 5 +``` + +The point of affinity is **isolation**. If your real-time pool owns cores 2-5 and your +background pool is confined to cores 0-1, the OS will not schedule background work onto the +cores running your control loop. Combined with a `Fifo` policy, this is how you keep jitter +out of the critical path. + +If you omit `affinity`, the OS is free to run the workers on any core. + +## Reserved pools: `background` and `rt` + +Two pool names have special meaning. + +### `background` + +When a source or task is marked `background: true`, it runs on the pool named `background`. +If you do not declare one, Copper creates a default `background` pool for you. Declare it +explicitly when you want to size it or pin it: + +```ron +runtime: ( + thread_pools: [ + ( id: "background", threads: 2, affinity: [0, 1] ), + ], +), +``` + +A task can also opt into a **different** pool with the object form of `background`: + +```ron +( + id: "detector", + type: "vision::Detector", + background: (pool: "vision"), +), +``` + +This runs `detector` on the `vision` pool instead of `background` -- useful when you want a +class of heavy tasks scheduled separately from the rest of your background work. See +[The Task Graph](./task-graph.md) for the `background` field, and +[Performance Basics](./performance-basics.md) for when backgrounding a stage is the right +call. + +### `rt` + +The pool named `rt` drives the [`parallel-rt`](./performance-basics.md) execution engine. +`parallel-rt` spawns one worker per generated process stage; defining an `rt` pool lets you +pin and prioritize those workers exactly like any other pool: + +```ron +runtime: ( + thread_pools: [ + ( id: "rt", threads: 4, affinity: [2, 3, 4, 5], policy: Fifo(priority: 80) ), + ], +), +``` + +The `rt` pool is consumed directly by the engine: its `affinity` and `policy` are applied +to the stage workers (Spread across the stage index), so unlike other pools you do not bind +tasks to it and you cannot use it as a `background` target. + +## When it cannot be applied: `on_error` + +Setting CPU affinity or a real-time priority can fail -- most commonly a `Fifo` priority +without `CAP_SYS_NICE`, or affinity to a core that does not exist. `on_error` chooses what +happens: + +- **`Warn`** (default) -- log a warning and fall back to default scheduling. The pool still + runs; it just does not get the requested affinity/priority. This keeps unprivileged + dev/laptop runs working out of the box. +- **`Strict`** -- fail at startup. Use this on a deployed real-time robot, where running the + control loop at default priority is a fault you want to catch loudly rather than discover + as jitter in the field. + +```ron +( id: "control", threads: 2, policy: Fifo(priority: 90), on_error: Strict ), +``` + +## The `rt-scheduling` feature + +Affinity and scheduling are gated behind the `rt-scheduling` Cargo feature on `cu29`, +forwarded from your crate the same way as the other runtime features: + +```toml +[features] +rt-scheduling = ["cu29/rt-scheduling"] +``` + +```bash +cargo run --features rt-scheduling +``` + +With the feature **off**, `runtime.thread_pools` still works -- pools are built with the +requested thread counts -- but `affinity` and `policy` are ignored and a warning is emitted. +This lets you keep one config that runs unprivileged in development and pins to real-time on +the target by flipping a feature flag. CPU affinity is cross-platform; the real-time +policies are Linux-only. + +## Good fit / Bad fit + +Good fit: + +```text +core 0-1: background pool (logging, telemetry) Nice / Fair +core 2-5: rt pool (control pipeline) Fifo, pinned +``` + +Why: + +- the critical pipeline owns dedicated cores at real-time priority +- everything else is confined elsewhere and yields to it + +Bad fit: + +```text +( id: "everything", threads: 8, policy: Fifo(priority: 99) ) +``` + +Why: + +- a single high-priority `Fifo` pool with no affinity, sized to every core, can starve the + OS itself +- real-time priority is for isolating *specific* work, not for "make it all fast" + +## Summary + +| Field | Purpose | Default | +|---|---|---| +| `id` | Pool name; `background` and `rt` are reserved | -- (required) | +| `threads` | Number of worker threads | -- (required) | +| `affinity` | CPU cores the workers are pinned to (Spread) | unpinned | +| `policy` | OS scheduling policy/priority | `Fair` | +| `on_error` | Behavior when affinity/policy cannot be applied | `Warn` | + +For complete working examples, see +[`examples/cu_background_task`](https://github.com/copper-project/copper-rs/tree/master/examples/cu_background_task) +(both `background` forms) and +[`examples/cu_runtime_matrix`](https://github.com/copper-project/copper-rs/tree/master/examples/cu_runtime_matrix) +(`rt` and `background` pools). diff --git a/book/src/troubleshooting-performance.md b/book/src/troubleshooting-performance.md index b7c0351..4ebe15d 100644 --- a/book/src/troubleshooting-performance.md +++ b/book/src/troubleshooting-performance.md @@ -244,7 +244,7 @@ Use this when **one isolated source or task** is too slow, but it does not need What it means semantically: -- Copper runs that source or task on the background threadpool +- Copper runs that source or task on the `background` thread pool - while it is still busy, `process()` returns `None` - downstream tasks therefore see missing output for some cycles @@ -268,9 +268,9 @@ Why: - the controller path usually needs one coherent output per cycle -Copper will ensure a `threadpool` resource bundle exists when background sources or tasks are present. -If you need a specific sizing, provide that bundle explicitly instead of relying on the -default. +If you do not declare a `background` pool, Copper creates a default one. To size it, pin it +to specific cores, or run this stage on a separate pool, see +[Thread Pools, Affinity, and Real-Time Scheduling](./thread-pools.md). ## Parallelize Inside One Task @@ -285,7 +285,7 @@ Typical cases: Preferred approach: - keep the DAG simple -- add a thread pool in resources or use a controlled parallel loop inside the task +- use a controlled parallel loop inside the task, or a [declared thread pool](./thread-pools.md) - measure the task again in the LAT tab Good fit: From bbe57da2fa0ff59ec74c65cb49804f4bd601dc81 Mon Sep 17 00:00:00 2001 From: Scofield626 Date: Tue, 16 Jun 2026 15:34:52 +0000 Subject: [PATCH 2/4] fix: clarify rt --- book/src/thread-pools.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/book/src/thread-pools.md b/book/src/thread-pools.md index 2b35b86..34e40d4 100644 --- a/book/src/thread-pools.md +++ b/book/src/thread-pools.md @@ -122,9 +122,11 @@ call. ### `rt` -The pool named `rt` drives the [`parallel-rt`](./performance-basics.md) execution engine. -`parallel-rt` spawns one worker per generated process stage; defining an `rt` pool lets you -pin and prioritize those workers exactly like any other pool: +`rt` is short for **real-time**, the same `rt` as in the `parallel-rt` feature and the +`rt-scheduling` feature -- it is not a name you picked, but a reserved id the runtime looks +for. The pool named `rt` drives the [`parallel-rt`](./performance-basics.md) execution +engine. `parallel-rt` spawns one worker per generated process stage; defining an `rt` pool +lets you pin and prioritize those workers exactly like any other pool: ```ron runtime: ( From 23e5dafbb0dcb16b2452a3db50f1f959e2e61f77 Mon Sep 17 00:00:00 2001 From: Scofield626 Date: Wed, 17 Jun 2026 16:35:58 +0000 Subject: [PATCH 3/4] docs(thread-pools): align with actual rt-scheduling behavior - Nice is Linux-only too: on non-Linux only `Fair` is applied; `Nice`, `Fifo`, and `RoundRobin` all fall back per `on_error`. - Note that negative niceness needs `CAP_SYS_NICE` like the RT policies. - State the default `background` pool size (2 threads, no affinity, Fair). - Call out that unknown pool names and `(pool: "rt")` on a background task are compile-time errors. - For the reserved `rt` pool, clarify that `parallel-rt` sizes workers by stage count, so `threads:` is ignored -- size `affinity` instead. - Add a "Determinism and replay" section: pool config is performance-only, the CopperList stream is bit-identical with/without an `rt` pool, and mutating `config.runtime.thread_pools` is the supported neutralization hook for replay/resim on hosts that can't pin. --- .gitignore | 1 + book/src/thread-pools.md | 45 +++++++++++++++++++++++++++++++--------- 2 files changed, 36 insertions(+), 10 deletions(-) diff --git a/.gitignore b/.gitignore index 1aa2975..fa295a1 100644 --- a/.gitignore +++ b/.gitignore @@ -21,3 +21,4 @@ Thumbs.db *~ .vscode/ .idea/ +.dmux/ diff --git a/book/src/thread-pools.md b/book/src/thread-pools.md index 34e40d4..aaca823 100644 --- a/book/src/thread-pools.md +++ b/book/src/thread-pools.md @@ -61,9 +61,11 @@ map directly onto the POSIX scheduling policies. > of the system. Use them for genuinely time-critical work, and pin them. Real-time policies are **Linux-only** and usually require the `CAP_SYS_NICE` capability (run -as root, grant the capability, or raise the limit in `/etc/security/limits.conf`). On other -platforms only `Fair` and `Nice` apply; requesting `Fifo` / `RoundRobin` fails according to -the pool's `on_error` setting. +as root, grant the capability, or raise the limit in `/etc/security/limits.conf`). Negative +niceness (`Nice(-1)` and below) needs the same capability -- without it the request falls +back per the pool's `on_error`. On non-Linux hosts only `Fair` is applied directly; `Nice`, +`Fifo`, and `RoundRobin` all fail according to the pool's `on_error`. CPU affinity is +cross-platform regardless. ## CPU affinity @@ -93,8 +95,9 @@ Two pool names have special meaning. ### `background` When a source or task is marked `background: true`, it runs on the pool named `background`. -If you do not declare one, Copper creates a default `background` pool for you. Declare it -explicitly when you want to size it or pin it: +If you do not declare one, Copper creates a default `background` pool for you: 2 worker +threads, no affinity, `Fair` scheduling. Declare it explicitly when you want to size it or +pin it: ```ron runtime: ( @@ -115,8 +118,12 @@ A task can also opt into a **different** pool with the object form of `backgroun ``` This runs `detector` on the `vision` pool instead of `background` -- useful when you want a -class of heavy tasks scheduled separately from the rest of your background work. See -[The Task Graph](./task-graph.md) for the `background` field, and +class of heavy tasks scheduled separately from the rest of your background work. The pool +name is resolved at build time: referencing a pool that isn't declared in +`runtime.thread_pools`, or pointing a background task at the reserved `rt` pool, is a +compile-time error. + +See [The Task Graph](./task-graph.md) for the `background` field, and [Performance Basics](./performance-basics.md) for when backgrounding a stage is the right call. @@ -136,9 +143,12 @@ runtime: ( ), ``` -The `rt` pool is consumed directly by the engine: its `affinity` and `policy` are applied -to the stage workers (Spread across the stage index), so unlike other pools you do not bind -tasks to it and you cannot use it as a `background` target. +The `rt` pool is consumed directly by the engine: only its `affinity` and `policy` are +applied to the stage workers (Spread across the stage index). `parallel-rt` spawns one +worker per generated process stage, so the `threads:` field has no effect on the `rt` pool +-- size `affinity` to match the stage count if you want exact one-core-per-stage pinning. +Unlike other pools, you do not bind tasks to it and you cannot use it as a `background` +target. ## When it cannot be applied: `on_error` @@ -177,6 +187,21 @@ This lets you keep one config that runs unprivileged in development and pins to the target by flipping a feature flag. CPU affinity is cross-platform; the real-time policies are Linux-only. +## Determinism and replay + +Pool configuration is **performance-only**. Affinity and policy change when and where work +runs, never what each task computes or in what order results commit to a copperlist. By +design, swapping in or out an `rt` pool produces a bit-identical normalized CopperList +stream in both sync and `parallel-rt` modes, and the runtime matrix integration tests guard +this as a regression. + +The practical consequence is that an offline replay or `resim` run on a different machine +keeps producing the same output as the live capture, even when the pool config cannot be +honored: with the default `on_error: Warn` the runtime logs and falls back to default +scheduling, and the stream is unchanged. If you need to *force* neutralization (for +example, to suppress warnings on a CI host that cannot pin, or to pick cores at runtime), +mutate `config.runtime.thread_pools` before building the app. + ## Good fit / Bad fit Good fit: From ced1f48b742a1a0a6b0dae5a70f52e6427c5d6c3 Mon Sep 17 00:00:00 2001 From: Scofield626 Date: Fri, 19 Jun 2026 09:37:16 +0000 Subject: [PATCH 4/4] fix: terminology and cores choosing --- book/src/advanced-tasks.md | 5 ++-- book/src/thread-pools.md | 50 +++++++++++++++++++++++++++++--------- 2 files changed, 40 insertions(+), 15 deletions(-) diff --git a/book/src/advanced-tasks.md b/book/src/advanced-tasks.md index 5811571..64939b8 100644 --- a/book/src/advanced-tasks.md +++ b/book/src/advanced-tasks.md @@ -111,9 +111,8 @@ The `type Resources<'r>` associated type declares what **hardware or system reso task needs. Resources represent physical endpoints -- serial ports, GPIO controllers, SPI buses, cameras -- or shared services bound through a resource bundle. -> Worker thread pools used to be declared as a resource. They now live in the `runtime` -> section of the config -- see -> [Thread Pools, Affinity, and Real-Time Scheduling](./thread-pools.md). +> Worker thread pools are configured separately, in the `runtime` section of the config +> -- see [Thread Pools, Affinity, and Real-Time Scheduling](./thread-pools.md). ### No resources diff --git a/book/src/thread-pools.md b/book/src/thread-pools.md index aaca823..6e0d949 100644 --- a/book/src/thread-pools.md +++ b/book/src/thread-pools.md @@ -40,10 +40,6 @@ Each pool has: - **`on_error`** (optional) -- what to do if the affinity or policy cannot be applied. Defaults to `Warn`. -> Earlier versions of Copper configured background threads through a `threadpool` -> **resource bundle**. That still works -- an existing `threadpool` resource is migrated -> automatically into a `background` pool -- but `runtime.thread_pools` is the way forward. - ## Scheduling policies The `policy` controls how the OS scheduler treats the pool's worker threads. On Linux these @@ -88,6 +84,34 @@ out of the critical path. If you omit `affinity`, the OS is free to run the workers on any core. +### Choosing cores + +Copper does not auto-detect kernel work or CPU topology -- the `affinity` list is taken at +face value. Two things on Linux are worth checking before you pick cores for a +latency-critical pool: + +- **Where the kernel handles IRQs.** Hardware interrupts (NIC, NVMe, USB, timers) run on + whichever CPUs the kernel binds them to, and a busy IRQ adds jitter to that core. + `cat /proc/interrupts` shows per-CPU interrupt counts; + `cat /proc/irq//smp_affinity_list` shows the affinity of a specific IRQ. By default + most distros either pin IRQs to CPU 0 or let `irqbalance` spread them; either way, the + conservative move is to keep your real-time `affinity` off the cores that show large + counts in `/proc/interrupts`. For full isolation, the standard tool is the kernel command + line (`isolcpus=`, `nohz_full=`, `rcu_nocbs=`), which removes the listed cores from the + general scheduler entirely. +- **Hyperthread siblings.** Two logical CPUs that share one physical core compete for the + same execution units, so pinning a real-time pool across a sibling pair gives almost no + isolation from background work landing on the other sibling. `lscpu -e` shows the + `CORE`/`CPU` mapping; `/sys/devices/system/cpu/cpuN/topology/thread_siblings_list` gives + the pair directly. The conservative move is to pick one logical CPU per physical core for + the `rt` pool and leave the siblings to `background` or to the OS. + +The explicit list is intentionally the lowest-level knob -- a list of logical CPU ids the +OS understands -- and there are no higher-level selectors (`BigCores`, `NumaNode`, "avoid +IRQs", "avoid siblings") yet. Future versions can layer those on top without changing the +list form, so the recommendation today is to inspect your target machine with the tools +above and write the list explicitly. + ## Reserved pools: `background` and `rt` Two pool names have special meaning. @@ -129,11 +153,11 @@ call. ### `rt` -`rt` is short for **real-time**, the same `rt` as in the `parallel-rt` feature and the -`rt-scheduling` feature -- it is not a name you picked, but a reserved id the runtime looks -for. The pool named `rt` drives the [`parallel-rt`](./performance-basics.md) execution -engine. `parallel-rt` spawns one worker per generated process stage; defining an `rt` pool -lets you pin and prioritize those workers exactly like any other pool: +The `rt` in `parallel-rt` stands for **runtime** -- the generated execution engine that +runs the task graph -- not "real-time". The pool named `rt` is the one the runtime looks +for to drive the [`parallel-rt`](./performance-basics.md) engine: it spawns one worker per +generated process stage, and the `rt` pool lets you pin and prioritize those workers +exactly like any other pool: ```ron runtime: ( @@ -152,9 +176,11 @@ target. ## When it cannot be applied: `on_error` -Setting CPU affinity or a real-time priority can fail -- most commonly a `Fifo` priority -without `CAP_SYS_NICE`, or affinity to a core that does not exist. `on_error` chooses what -happens: +Setting CPU affinity or a real-time priority can fail. The common cases are platform- +specific: on **Linux**, a `Fifo` / `RoundRobin` priority (or `Nice` below 0) without +`CAP_SYS_NICE`; on **Windows**, any non-`Fair` policy at all, since the POSIX real-time +policies are not implemented there; and on either OS, an `affinity` entry that names a +core the machine does not have. `on_error` chooses what happens: - **`Warn`** (default) -- log a warning and fall back to default scheduling. The pool still runs; it just does not get the requested affinity/priority. This keeps unprivileged