Skip to content

Commit 403b37b

Browse files
committed
feat(api): wire container healthcheck observer end to end
Implements the full healthcheck observer that populates `ContainerSnapshot.health` (the read-only field reserved by CHAOS-1319) by running the configured probe inside the running container, interpreting exit codes through a Docker-compatible state machine, and writing the result back through the `ContainersService` actor under a generation-gated update path. Motivation ---------- CHAOS-1319 reserved the SDK shape (`HealthStatus` enum + optional `health` field on `ContainerSnapshot`) but the daemon never populated it; the field is always `nil` today, so external orchestrators (the canonical use case is a compose-spec orchestrator implementing `depends_on.condition: service_healthy`) can only block on image-baked healthchecks and only when the underlying runtime owns the probe loop. Real workloads (databases that take seconds to accept connections, queue brokers that warm up an in-memory state) need a container-level healthcheck observer that the daemon owns. This PR adds it. What this PR changes -------------------- - Sources/ContainerResource/Container/Healthcheck.swift (new): public Codable / Sendable struct mirroring the Docker / compose-spec schema (`test`, `interval`, `timeout`, `retries`, `start_period`, `start_interval`, `disable`). Validates the probe shape (`NONE` / `CMD` / `CMD-SHELL`) and rejects malformed inputs with actionable error messages. - Sources/ContainerResource/Container/ContainerConfiguration.swift: new optional `healthcheck: Healthcheck?` field, `decodeIfPresent` on the wire so legacy on-disk configurations decode unchanged. - Sources/Services/ContainerAPIService/Server/Containers/ HealthStateMachine.swift (new): pure value type that maps probe outcomes to `HealthStatus`. Implements the Docker-compatible flow: initial `.starting`, immediate transition to `.healthy` on the first successful probe (including during the `start_period` grace window), `retries` consecutive failures post-grace transition to `.unhealthy`, recovery to `.healthy` without restart. - Sources/Services/ContainerAPIService/Server/Containers/ HealthProber.swift (new): `HealthProber` protocol plus production `SandboxClientHealthProber` that drives an existing `SandboxClient` to spawn a fresh `__container_healthcheck_<UUID>` synthetic process per probe, races `wait()` against a per-probe timeout, and signals `SIGKILL` on timeout to unblock the synthetic wait task before draining the task group. - Sources/Services/ContainerAPIService/Server/Containers/ HealthMonitor.swift (new): per-container observer manager actor that mirrors `ExitMonitor`. `register(id:generation:startedAt: healthcheck:prober:onUpdate:)` cancels any prior observer, fires the initial `.starting` (or `.none` for disabled checks) callback, and runs the probe loop. `unregister(id:)` is idempotent and triggers cooperative cancellation. - Sources/Services/ContainerAPIService/Server/Containers/ ContainersService.swift: new private `healthMonitor: HealthMonitor` field; new `healthGeneration: UInt64` token on `ContainerState` bumped on every transition into `.running`; observer registered inside `startProcess` once the init process is up; unregister wired into `handleContainerExit`. New private `applyHealthUpdate(id: generation:status:)` is the single mutation entry; it drops updates whose generation no longer matches the live container or whose status is no longer `.running`, closing the late-callback / restart race. - Sources/Services/ContainerAPIService/Client/Flags.swift: seven new flags on `Flags.Management` covering `--health-cmd`, `--health-interval`, `--health-timeout`, `--health-retries`, `--health-start-period`, `--health-start-interval`, and `--no-healthcheck`. - Sources/Services/ContainerAPIService/Client/Utility.swift: new private `makeHealthcheck(management:)` that translates the flag bag into a `Healthcheck`. Rejects orphan `--health-*` flags without `--health-cmd` to catch typos at submit time. - Package.swift: `ContainerAPIServiceTests` gains a dependency on the `ContainerAPIService` target so the new tests can use the `@testable` import. - Tests: - Tests/ContainerResourceTests/HealthcheckTest.swift: 12 tests covering shape parsing (`CMD` / `CMD-SHELL` / `NONE`), validation error paths, the `disable` flag, the `probeInterval` selection rule (start-interval inside the grace window only), and a legacy-config Codable round-trip regression. - Tests/ContainerAPIServiceTests/HealthStateMachineTest.swift: 10 tests exercising every transition documented in the design: initial state, success during grace, failure during grace, failures past grace toward `retries`, success resets the counter, `unhealthy` recovers without restart, disabled machine ignores inputs, retries=0 corner case. - Tests/ContainerAPIServiceTests/HealthMonitorTest.swift: 4 tests against a `ScriptedProber` actor (deterministic probe outcomes) and a `StatusRecorder` (ordered update capture). Covers the disabled-check single-callback path, the `.starting` -> `.healthy` transition, the consecutive-failure -> `.unhealthy` path, and the unregister-cancels-loop guarantee. Design notes ------------ The implementation follows the architecture recommendation produced during a design consult (see CHAOS-1381 thread): observer placement in a dedicated actor (mirroring `ExitMonitor`), probe execution through the existing `createProcess` / `startProcess` / `wait` path (no new XPC route added), Docker-compatible state machine semantics, and generation-gated snapshot updates rather than relying on cancellation alone to suppress stale callbacks. Wire compatibility ------------------ `ContainerConfiguration.healthcheck` is a new optional field, decoded with `decodeIfPresent`. Containers persisted by older daemons round-trip cleanly (covered by `testLegacyContainerConfigurationDecodesWithoutHealthcheck`). New CLI flags are independent and have no effect when omitted, so older clients hitting a newer daemon and vice versa both behave identically to today. Known limitations (intentional, follow-up work) ----------------------------------------------- - The `--health-cmd` CLI shape currently accepts only the shell form (translated to `["CMD-SHELL", cmd]`). The richer `["CMD", "exec", "arg1", ...]` form is reachable via API clients that build `Healthcheck` directly (e.g. compose orchestrators). Adding a CLI surface for CMD-form probes is a follow-up. - Daemon restart does not rehydrate health state. On daemon launch, observers are restarted from `.starting` rather than persisting probe counters. Per the design consult this is deliberate scope for v1. - Probe intervals use Foundation `TimeInterval` (Double seconds). Compose-spec duration strings (`30s`, `1m30s`) are parsed by the client (e.g. container-compose) before reaching the API. Pairs with CHAOS-1319 --------------------- CHAOS-1319 reserved the SDK shape (`ContainerSnapshot.health`). This PR is the runtime that populates it, closing the loop for compose-spec `depends_on.condition: service_healthy` against container-compose orchestrators. CHAOS-1319's PR (#13) should land first or be batched with this one. Verification ------------ - `swift build -c release` clean on macOS 26 / Apple silicon. - `swift test --filter 'HealthcheckTest|HealthStateMachineTest| HealthMonitorTest'` passes 26/26: 12 Healthcheck data shape + Codable + validation, 10 pure HealthStateMachine transitions, 4 HealthMonitor actor lifecycle / cancellation tests.
1 parent 005536a commit 403b37b

12 files changed

Lines changed: 1125 additions & 0 deletions

File tree

Package.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,7 @@ let package = Package(
203203
name: "ContainerAPIServiceTests",
204204
dependencies: [
205205
.product(name: "Containerization", package: "containerization"),
206+
"ContainerAPIService",
206207
"ContainerResource",
207208
"ContainerRuntimeLinuxClient",
208209
"ContainerRuntimeClient",

Sources/ContainerResource/Container/ContainerConfiguration.swift

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,10 @@ public struct ContainerConfiguration: Sendable, Codable {
6161
public var shmSize: UInt64?
6262
/// Signal to send to the container process on stop (from image config).
6363
public var stopSignal: String?
64+
/// Optional periodic healthcheck spec. When set and not effectively
65+
/// disabled, the API server starts a per-container observer that runs
66+
/// the configured probe and updates ``ContainerSnapshot/health``.
67+
public var healthcheck: Healthcheck?
6468

6569
enum CodingKeys: String, CodingKey {
6670
case id
@@ -85,6 +89,7 @@ public struct ContainerConfiguration: Sendable, Codable {
8589
case capDrop
8690
case shmSize
8791
case stopSignal
92+
case healthcheck
8893
}
8994

9095
/// Create a configuration from the supplied Decoder, initializing missing
@@ -120,6 +125,7 @@ public struct ContainerConfiguration: Sendable, Codable {
120125
capDrop = try container.decodeIfPresent([String].self, forKey: .capDrop) ?? []
121126
shmSize = try container.decodeIfPresent(UInt64.self, forKey: .shmSize)
122127
stopSignal = try container.decodeIfPresent(String.self, forKey: .stopSignal)
128+
healthcheck = try container.decodeIfPresent(Healthcheck.self, forKey: .healthcheck)
123129
}
124130

125131
public struct DNSConfiguration: Sendable, Codable {
Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
//===----------------------------------------------------------------------===//
2+
// Copyright © 2026 Apple Inc. and the container project authors.
3+
//
4+
// Licensed under the Apache License, Version 2.0 (the "License");
5+
// you may not use this file except in compliance with the License.
6+
// You may obtain a copy of the License at
7+
//
8+
// https://www.apache.org/licenses/LICENSE-2.0
9+
//
10+
// Unless required by applicable law or agreed to in writing, software
11+
// distributed under the License is distributed on an "AS IS" BASIS,
12+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
// See the License for the specific language governing permissions and
14+
// limitations under the License.
15+
//===----------------------------------------------------------------------===//
16+
17+
import ContainerizationError
18+
import Foundation
19+
20+
/// Configuration for a periodic, container-level healthcheck.
21+
///
22+
/// The shape mirrors the Docker / compose-spec healthcheck schema so that
23+
/// downstream tools (the canonical use case is a compose-spec orchestrator
24+
/// implementing `depends_on.condition: service_healthy`) can populate this
25+
/// type directly from a `docker-compose.yml` `healthcheck:` block.
26+
///
27+
/// Semantics applied by the daemon's healthcheck observer:
28+
///
29+
/// 1. When the observer starts and the healthcheck is enabled, the
30+
/// container's ``ContainerSnapshot/health`` is set to
31+
/// ``HealthStatus/starting``.
32+
/// 2. While the wall-clock age of the container is within ``startPeriod``,
33+
/// failed probes do not advance the consecutive failure counter.
34+
/// Successful probes during the grace period transition the container
35+
/// immediately to ``HealthStatus/healthy``.
36+
/// 3. After the grace period elapses, ``retries`` consecutive failed probes
37+
/// transition the container to ``HealthStatus/unhealthy``. A subsequent
38+
/// successful probe resets the counter and transitions back to
39+
/// ``HealthStatus/healthy`` without requiring a restart.
40+
/// 4. A probe that does not return within ``timeout`` counts as a failed
41+
/// probe.
42+
/// 5. ``test`` of `["NONE"]` and ``disable`` set to `true` both bypass the
43+
/// observer entirely; ``ContainerSnapshot/health`` remains `nil`.
44+
public struct Healthcheck: Codable, Sendable, Equatable {
45+
/// The probe specification.
46+
///
47+
/// Compatible shapes:
48+
/// - `["NONE"]` — disable any healthcheck inherited from the image.
49+
/// - `["CMD", "executable", "arg1", ...]` — run `executable` with the
50+
/// supplied arguments directly inside the container. Exit code `0`
51+
/// means healthy, any other exit code means unhealthy.
52+
/// - `["CMD-SHELL", "shell command string"]` — run the entire command
53+
/// string through the container's default shell (`/bin/sh -c`).
54+
public let test: [String]
55+
56+
/// Time between consecutive probes, in seconds. Defaults to 30 seconds.
57+
public let interval: TimeInterval
58+
59+
/// Per-probe deadline, in seconds. A probe that does not return within
60+
/// this window counts as a failed probe. Defaults to 30 seconds.
61+
public let timeout: TimeInterval
62+
63+
/// Number of consecutive failed probes that transition the container
64+
/// from ``HealthStatus/healthy`` (or ``HealthStatus/starting``) to
65+
/// ``HealthStatus/unhealthy``. Defaults to 3.
66+
public let retries: Int
67+
68+
/// Optional grace window, in seconds, during which failed probes do not
69+
/// count toward ``retries``. The first successful probe during this
70+
/// window transitions the container immediately to
71+
/// ``HealthStatus/healthy``. When `nil`, no grace is applied.
72+
public let startPeriod: TimeInterval?
73+
74+
/// Optional probe interval used while the container is still within
75+
/// ``startPeriod``. When `nil`, ``interval`` is used during the grace
76+
/// window as well.
77+
public let startInterval: TimeInterval?
78+
79+
/// Bypass the observer entirely. Equivalent to ``test`` = `["NONE"]`.
80+
public let disable: Bool?
81+
82+
/// Default probe interval applied when the configuration omits one.
83+
public static let defaultInterval: TimeInterval = 30
84+
/// Default per-probe deadline applied when the configuration omits one.
85+
public static let defaultTimeout: TimeInterval = 30
86+
/// Default consecutive-failure threshold applied when the configuration
87+
/// omits one.
88+
public static let defaultRetries: Int = 3
89+
90+
public init(
91+
test: [String],
92+
interval: TimeInterval = Healthcheck.defaultInterval,
93+
timeout: TimeInterval = Healthcheck.defaultTimeout,
94+
retries: Int = Healthcheck.defaultRetries,
95+
startPeriod: TimeInterval? = nil,
96+
startInterval: TimeInterval? = nil,
97+
disable: Bool? = nil
98+
) throws {
99+
self.test = test
100+
self.interval = interval
101+
self.timeout = timeout
102+
self.retries = retries
103+
self.startPeriod = startPeriod
104+
self.startInterval = startInterval
105+
self.disable = disable
106+
try validate()
107+
}
108+
109+
enum CodingKeys: String, CodingKey {
110+
case test
111+
case interval
112+
case timeout
113+
case retries
114+
case startPeriod
115+
case startInterval
116+
case disable
117+
}
118+
119+
public init(from decoder: Decoder) throws {
120+
let container = try decoder.container(keyedBy: CodingKeys.self)
121+
test = try container.decode([String].self, forKey: .test)
122+
interval = try container.decodeIfPresent(TimeInterval.self, forKey: .interval) ?? Healthcheck.defaultInterval
123+
timeout = try container.decodeIfPresent(TimeInterval.self, forKey: .timeout) ?? Healthcheck.defaultTimeout
124+
retries = try container.decodeIfPresent(Int.self, forKey: .retries) ?? Healthcheck.defaultRetries
125+
startPeriod = try container.decodeIfPresent(TimeInterval.self, forKey: .startPeriod)
126+
startInterval = try container.decodeIfPresent(TimeInterval.self, forKey: .startInterval)
127+
disable = try container.decodeIfPresent(Bool.self, forKey: .disable)
128+
try validate()
129+
}
130+
131+
public func encode(to encoder: Encoder) throws {
132+
var container = encoder.container(keyedBy: CodingKeys.self)
133+
try container.encode(test, forKey: .test)
134+
try container.encode(interval, forKey: .interval)
135+
try container.encode(timeout, forKey: .timeout)
136+
try container.encode(retries, forKey: .retries)
137+
try container.encodeIfPresent(startPeriod, forKey: .startPeriod)
138+
try container.encodeIfPresent(startInterval, forKey: .startInterval)
139+
try container.encodeIfPresent(disable, forKey: .disable)
140+
}
141+
142+
/// Whether the healthcheck is effectively disabled (no observer should
143+
/// be started, ``ContainerSnapshot/health`` remains `nil`).
144+
public var isEffectivelyDisabled: Bool {
145+
if disable == true { return true }
146+
if test.count == 1 && test[0] == "NONE" { return true }
147+
return false
148+
}
149+
150+
/// The probe interval that should be used at the supplied wall-clock age
151+
/// of the container. Returns ``startInterval`` while the container is
152+
/// still within ``startPeriod``, otherwise ``interval``.
153+
public func probeInterval(forContainerAge age: TimeInterval) -> TimeInterval {
154+
if let startPeriod, age < startPeriod, let startInterval {
155+
return startInterval
156+
}
157+
return interval
158+
}
159+
160+
private func validate() throws {
161+
guard !test.isEmpty else {
162+
throw ContainerizationError(
163+
.invalidArgument,
164+
message: "healthcheck test must not be empty"
165+
)
166+
}
167+
if !isEffectivelyDisabled {
168+
switch test[0] {
169+
case "CMD", "CMD-SHELL":
170+
guard test.count >= 2 else {
171+
throw ContainerizationError(
172+
.invalidArgument,
173+
message: "healthcheck test '\(test[0])' requires at least one argument"
174+
)
175+
}
176+
default:
177+
throw ContainerizationError(
178+
.invalidArgument,
179+
message: "healthcheck test must start with 'NONE', 'CMD', or 'CMD-SHELL' (got '\(test[0])')"
180+
)
181+
}
182+
}
183+
guard interval > 0 else {
184+
throw ContainerizationError(
185+
.invalidArgument,
186+
message: "healthcheck interval must be positive (got \(interval))"
187+
)
188+
}
189+
guard timeout > 0 else {
190+
throw ContainerizationError(
191+
.invalidArgument,
192+
message: "healthcheck timeout must be positive (got \(timeout))"
193+
)
194+
}
195+
guard retries >= 0 else {
196+
throw ContainerizationError(
197+
.invalidArgument,
198+
message: "healthcheck retries must be non-negative (got \(retries))"
199+
)
200+
}
201+
if let startPeriod, startPeriod < 0 {
202+
throw ContainerizationError(
203+
.invalidArgument,
204+
message: "healthcheck start_period must be non-negative (got \(startPeriod))"
205+
)
206+
}
207+
if let startInterval, startInterval <= 0 {
208+
throw ContainerizationError(
209+
.invalidArgument,
210+
message: "healthcheck start_interval must be positive (got \(startInterval))"
211+
)
212+
}
213+
}
214+
}

Sources/Services/ContainerAPIService/Client/Flags.swift

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -345,6 +345,48 @@ public struct Flags {
345345
@Option(name: [.customLong("volume"), .short], help: "Bind mount a volume into the container")
346346
public var volumes: [String] = []
347347

348+
@Option(
349+
name: .customLong("health-cmd"),
350+
help: "Healthcheck command to run inside the container (executed via /bin/sh -c)."
351+
)
352+
public var healthCmd: String?
353+
354+
@Option(
355+
name: .customLong("health-interval"),
356+
help: "Time between healthcheck probes, in seconds (default 30)."
357+
)
358+
public var healthInterval: Double?
359+
360+
@Option(
361+
name: .customLong("health-timeout"),
362+
help: "Per-probe deadline for the healthcheck, in seconds (default 30)."
363+
)
364+
public var healthTimeout: Double?
365+
366+
@Option(
367+
name: .customLong("health-retries"),
368+
help: "Number of consecutive failed probes before the container is reported unhealthy (default 3)."
369+
)
370+
public var healthRetries: Int?
371+
372+
@Option(
373+
name: .customLong("health-start-period"),
374+
help: "Grace window after start during which failed probes do not count, in seconds."
375+
)
376+
public var healthStartPeriod: Double?
377+
378+
@Option(
379+
name: .customLong("health-start-interval"),
380+
help: "Probe interval used while still within the grace window, in seconds."
381+
)
382+
public var healthStartInterval: Double?
383+
384+
@Flag(
385+
name: .customLong("no-healthcheck"),
386+
help: "Disable any image-baked healthcheck for this container."
387+
)
388+
public var noHealthcheck: Bool = false
389+
348390
public func validate() throws {
349391
if dnsDisabled {
350392
let hasDNSConfig =

Sources/Services/ContainerAPIService/Client/Utility.swift

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,9 +268,38 @@ public struct Utility {
268268
config.runtimeHandler = runtime
269269
}
270270

271+
config.healthcheck = try Self.makeHealthcheck(management: management)
272+
271273
return (config, kernel, management.initImage)
272274
}
273275

276+
private static func makeHealthcheck(management: Flags.Management) throws -> Healthcheck? {
277+
if management.noHealthcheck {
278+
return try Healthcheck(test: ["NONE"])
279+
}
280+
guard let cmd = management.healthCmd else {
281+
// Reject orphan health-* flags without a command — catch typos early.
282+
if management.healthInterval != nil || management.healthTimeout != nil
283+
|| management.healthRetries != nil || management.healthStartPeriod != nil
284+
|| management.healthStartInterval != nil
285+
{
286+
throw ContainerizationError(
287+
.invalidArgument,
288+
message: "--health-* flags require --health-cmd to be specified"
289+
)
290+
}
291+
return nil
292+
}
293+
return try Healthcheck(
294+
test: ["CMD-SHELL", cmd],
295+
interval: management.healthInterval ?? Healthcheck.defaultInterval,
296+
timeout: management.healthTimeout ?? Healthcheck.defaultTimeout,
297+
retries: management.healthRetries ?? Healthcheck.defaultRetries,
298+
startPeriod: management.healthStartPeriod,
299+
startInterval: management.healthStartInterval
300+
)
301+
}
302+
274303
static func getAttachmentConfigurations(
275304
containerId: String,
276305
builtinNetworkId: String?,

0 commit comments

Comments
 (0)