Commit 403b37b
committed
feat(api): wire container healthcheck observer end to end
Implements the full healthcheck observer that populates
`ContainerSnapshot.health` (the read-only field reserved by CHAOS-1319)
by running the configured probe inside the running container,
interpreting exit codes through a Docker-compatible state machine, and
writing the result back through the `ContainersService` actor under a
generation-gated update path.
Motivation
----------
CHAOS-1319 reserved the SDK shape (`HealthStatus` enum + optional
`health` field on `ContainerSnapshot`) but the daemon never populated
it; the field is always `nil` today, so external orchestrators (the
canonical use case is a compose-spec orchestrator implementing
`depends_on.condition: service_healthy`) can only block on
image-baked healthchecks and only when the underlying runtime owns
the probe loop. Real workloads (databases that take seconds to accept
connections, queue brokers that warm up an in-memory state) need a
container-level healthcheck observer that the daemon owns. This PR
adds it.
What this PR changes
--------------------
- Sources/ContainerResource/Container/Healthcheck.swift (new):
public Codable / Sendable struct mirroring the Docker / compose-spec
schema (`test`, `interval`, `timeout`, `retries`, `start_period`,
`start_interval`, `disable`). Validates the probe shape (`NONE` /
`CMD` / `CMD-SHELL`) and rejects malformed inputs with actionable
error messages.
- Sources/ContainerResource/Container/ContainerConfiguration.swift:
new optional `healthcheck: Healthcheck?` field, `decodeIfPresent`
on the wire so legacy on-disk configurations decode unchanged.
- Sources/Services/ContainerAPIService/Server/Containers/
HealthStateMachine.swift (new): pure value type that maps probe
outcomes to `HealthStatus`. Implements the Docker-compatible flow:
initial `.starting`, immediate transition to `.healthy` on the
first successful probe (including during the `start_period` grace
window), `retries` consecutive failures post-grace transition to
`.unhealthy`, recovery to `.healthy` without restart.
- Sources/Services/ContainerAPIService/Server/Containers/
HealthProber.swift (new): `HealthProber` protocol plus production
`SandboxClientHealthProber` that drives an existing `SandboxClient`
to spawn a fresh `__container_healthcheck_<UUID>` synthetic process
per probe, races `wait()` against a per-probe timeout, and signals
`SIGKILL` on timeout to unblock the synthetic wait task before
draining the task group.
- Sources/Services/ContainerAPIService/Server/Containers/
HealthMonitor.swift (new): per-container observer manager actor
that mirrors `ExitMonitor`. `register(id:generation:startedAt:
healthcheck:prober:onUpdate:)` cancels any prior observer, fires
the initial `.starting` (or `.none` for disabled checks) callback,
and runs the probe loop. `unregister(id:)` is idempotent and
triggers cooperative cancellation.
- Sources/Services/ContainerAPIService/Server/Containers/
ContainersService.swift: new private `healthMonitor: HealthMonitor`
field; new `healthGeneration: UInt64` token on `ContainerState`
bumped on every transition into `.running`; observer registered
inside `startProcess` once the init process is up; unregister wired
into `handleContainerExit`. New private `applyHealthUpdate(id:
generation:status:)` is the single mutation entry; it drops updates
whose generation no longer matches the live container or whose
status is no longer `.running`, closing the late-callback /
restart race.
- Sources/Services/ContainerAPIService/Client/Flags.swift: seven new
flags on `Flags.Management` covering `--health-cmd`,
`--health-interval`, `--health-timeout`, `--health-retries`,
`--health-start-period`, `--health-start-interval`, and
`--no-healthcheck`.
- Sources/Services/ContainerAPIService/Client/Utility.swift: new
private `makeHealthcheck(management:)` that translates the flag
bag into a `Healthcheck`. Rejects orphan `--health-*` flags
without `--health-cmd` to catch typos at submit time.
- Package.swift: `ContainerAPIServiceTests` gains a dependency on
the `ContainerAPIService` target so the new tests can use the
`@testable` import.
- Tests:
- Tests/ContainerResourceTests/HealthcheckTest.swift: 12 tests
covering shape parsing (`CMD` / `CMD-SHELL` / `NONE`), validation
error paths, the `disable` flag, the `probeInterval` selection
rule (start-interval inside the grace window only), and a
legacy-config Codable round-trip regression.
- Tests/ContainerAPIServiceTests/HealthStateMachineTest.swift: 10
tests exercising every transition documented in the design:
initial state, success during grace, failure during grace,
failures past grace toward `retries`, success resets the counter,
`unhealthy` recovers without restart, disabled machine ignores
inputs, retries=0 corner case.
- Tests/ContainerAPIServiceTests/HealthMonitorTest.swift: 4 tests
against a `ScriptedProber` actor (deterministic probe outcomes)
and a `StatusRecorder` (ordered update capture). Covers the
disabled-check single-callback path, the `.starting` -> `.healthy`
transition, the consecutive-failure -> `.unhealthy` path, and
the unregister-cancels-loop guarantee.
Design notes
------------
The implementation follows the architecture recommendation produced
during a design consult (see CHAOS-1381 thread): observer placement
in a dedicated actor (mirroring `ExitMonitor`), probe execution
through the existing `createProcess` / `startProcess` / `wait` path
(no new XPC route added), Docker-compatible state machine semantics,
and generation-gated snapshot updates rather than relying on
cancellation alone to suppress stale callbacks.
Wire compatibility
------------------
`ContainerConfiguration.healthcheck` is a new optional field,
decoded with `decodeIfPresent`. Containers persisted by older
daemons round-trip cleanly (covered by
`testLegacyContainerConfigurationDecodesWithoutHealthcheck`). New
CLI flags are independent and have no effect when omitted, so older
clients hitting a newer daemon and vice versa both behave
identically to today.
Known limitations (intentional, follow-up work)
-----------------------------------------------
- The `--health-cmd` CLI shape currently accepts only the shell
form (translated to `["CMD-SHELL", cmd]`). The richer
`["CMD", "exec", "arg1", ...]` form is reachable via API clients
that build `Healthcheck` directly (e.g. compose orchestrators).
Adding a CLI surface for CMD-form probes is a follow-up.
- Daemon restart does not rehydrate health state. On daemon launch,
observers are restarted from `.starting` rather than persisting
probe counters. Per the design consult this is deliberate scope
for v1.
- Probe intervals use Foundation `TimeInterval` (Double seconds).
Compose-spec duration strings (`30s`, `1m30s`) are parsed by the
client (e.g. container-compose) before reaching the API.
Pairs with CHAOS-1319
---------------------
CHAOS-1319 reserved the SDK shape (`ContainerSnapshot.health`).
This PR is the runtime that populates it, closing the loop for
compose-spec `depends_on.condition: service_healthy` against
container-compose orchestrators. CHAOS-1319's PR
(#13) should land first or be batched with
this one.
Verification
------------
- `swift build -c release` clean on macOS 26 / Apple silicon.
- `swift test --filter 'HealthcheckTest|HealthStateMachineTest|
HealthMonitorTest'` passes 26/26: 12 Healthcheck data shape +
Codable + validation, 10 pure HealthStateMachine transitions, 4
HealthMonitor actor lifecycle / cancellation tests.1 parent 005536a commit 403b37b
12 files changed
Lines changed: 1125 additions & 0 deletions
File tree
- Sources
- ContainerResource/Container
- Services/ContainerAPIService
- Client
- Server/Containers
- Tests
- ContainerAPIServiceTests
- ContainerResourceTests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
203 | 203 | | |
204 | 204 | | |
205 | 205 | | |
| 206 | + | |
206 | 207 | | |
207 | 208 | | |
208 | 209 | | |
| |||
Lines changed: 6 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
64 | 68 | | |
65 | 69 | | |
66 | 70 | | |
| |||
85 | 89 | | |
86 | 90 | | |
87 | 91 | | |
| 92 | + | |
88 | 93 | | |
89 | 94 | | |
90 | 95 | | |
| |||
120 | 125 | | |
121 | 126 | | |
122 | 127 | | |
| 128 | + | |
123 | 129 | | |
124 | 130 | | |
125 | 131 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
345 | 345 | | |
346 | 346 | | |
347 | 347 | | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
348 | 390 | | |
349 | 391 | | |
350 | 392 | | |
| |||
Lines changed: 29 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
268 | 268 | | |
269 | 269 | | |
270 | 270 | | |
| 271 | + | |
| 272 | + | |
271 | 273 | | |
272 | 274 | | |
273 | 275 | | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
274 | 303 | | |
275 | 304 | | |
276 | 305 | | |
| |||
0 commit comments