Repository navigation

Add firmware trace and direct C comparison to OLED evidence #159

Workflow file for this run

.github/workflows/build-and-test.yml at 2f6602e

	# The pipeline that compiles Swift. Without it, a green branch meant only that
	# somebody remembered to run `swift test` locally and read the output.
	#
	# The jobs:
	# test CandelaKit's suite, the engine's automated coverage
	# app-build the app target compiles, which the Kit suite cannot tell
	# you, then CandelaAppTests runs over the app's own sources
	# release-build the configuration that ships compiles, and carries none
	# of the debug switches added while iterating
	#
	# Why a hosted runner is a valid environment for the suite and not a degraded
	# one: CandelaKit is hardware-free by design, imports no AppKit or SwiftUI, and
	# needs no attached panel, no Accessibility grant and no signing identity.
	# Everything that touches a display sits behind a protocol and is faked in the
	# tests, so a machine with no monitors runs the whole suite rather than a subset
	# of it. Nothing here verifies display BEHAVIOUR, and nothing here ever will:
	# that is what candela-probe and the hardware passes are for.
	name: Build and test

	on:
	push:
	branches: [main]
	paths-ignore:
	- '**.md'
	- 'docs/**'
	- 'site/**'
	- 'LICENSE'
	- '.github/assets/**'
	- '.github/ISSUE_TEMPLATE/**'
	- '.github/PULL_REQUEST_TEMPLATE.md'
	- '.github/workflows/site.yml'
	# Every PR gets a result. Suite selection below avoids macOS work for docs and site changes.
	pull_request:
	branches: [main]
	workflow_dispatch:

	permissions:
	contents: read

	# PR runs supersede each other; main runs never do. Only a PR's tip has to be
	# green, but every commit on main is one `git bisect` can land on and one a
	# release can be cut from, so a run cancelled by the next push leaves that
	# commit with no verdict at all.
	#
	# The group key does that work, not cancel-in-progress. With the flag false and
	# every main push still sharing one group, GitHub cancels the queued `pending`
	# run when the next one arrives, so three quick pushes lose the middle commit's
	# verdict anyway and the fix looks like it worked. Non-PR runs therefore key on
	# the run id: unique per run, so nothing queues behind anything or cancels it.
	concurrency:
	group: build-and-test-${{ github.event_name == 'pull_request' && github.ref \|\| github.run_id }}
	cancel-in-progress: ${{ github.event_name == 'pull_request' }}

	jobs:
	scope:
	runs-on: ubuntu-latest
	timeout-minutes: 5
	outputs:
	app: ${{ steps.scope.outputs.app }}
	steps:
	- uses: actions/checkout@v4
	with:
	fetch-depth: 2
	- name: Test suite selection and result handling
	run: python3 tools/ci/test_verification.py
	- name: Select suites
	id: scope
	run: python3 tools/ci/verification.py scope >> "$GITHUB_OUTPUT"

	test:
	needs: scope
	if: needs.scope.outputs.app == 'true'
	# Pinned to macos-26, not macos-latest, and matched to the dev machine's
	# major version on purpose. The label decides which Swift compiler runs, and
	# a compiler older than the one the code is written against produces reds
	# that describe the runner rather than the diff. macos-15 carries Xcode 16
	# (Swift 6.0/6.1) against Xcode 26 locally, which is a real gap to gamble on
	# for no benefit. `latest` is the same gamble with the added property that it
	# moves without a commit.
	#
	# The cost of pinning is that this line needs a bump when the image is
	# retired, which GitHub announces and which fails loudly rather than
	# silently. That is the cheaper failure.
	runs-on: macos-26
	timeout-minutes: 20
	steps:
	- uses: actions/checkout@v4

	# Printed, never asserted. Pinning an exact Xcode build here would redden
	# the pipeline the day the image rolls forward, a standing maintenance
	# cost with no matching risk: the package declares swift-tools 6.0 and a
	# macOS 14 floor, so it builds on any toolchain new enough for both. What
	# this step buys is attribution. When a run fails for a reason the diff
	# does not explain, the first question asked is which toolchain it ran on,
	# and the answer should already be in the log.
	- name: Toolchain
	run: \|
	xcodebuild -version
	swift --version

	- name: Run the suite
	working-directory: CandelaKit
	shell: bash
	run: \|
	set -euo pipefail
	# pipefail is load-bearing here, not boilerplate. A pipeline reports
	# the exit status of its LAST command, so `swift test \| tee` without it
	# reports tee's status, and tee succeeds whatever swift test did. Every
	# failing suite would be published as green: a reported success that
	# was never achieved, one layer up from the hardware.
	swift test 2>&1 \| tee "$RUNNER_TEMP/kit-tests.log"

	# A gate nobody has seen fail is a gate nobody should trust. The failure
	# mode worth guarding is not a red build; it is a GREEN one that exercised
	# nothing, and reaching it takes no bug at all.
	# A test target dropped from Package.swift, a filter matching no test, a
	# build that produced an empty bundle: each exits 0 having run zero tests,
	# and the step above cannot tell that apart from a pass.
	#
	# So the run's own summary line is read back out of the log. Absence of
	# that line is a failure in its own right: it means either nothing ran or
	# the output format moved, and under both readings the zero-exit above has
	# stopped being evidence of anything.
	#
	# Both halves were demonstrated failing on this runner before the workflow
	# merged [MEASURED 2026-08-11], because a gate whose red path nobody has
	# watched is a gate nobody should trust:
	# run 31542488117 one expectation flipped to a wrong value. `swift test`
	# exited 1, the status survived the pipe into tee, and
	# the job went red at "Run the suite".
	# run 31542654491 the suite filtered down to a name matching nothing.
	# `swift test` printed "Test run with 0 tests in 0
	# suites passed" and exited 0, so "Run the suite" was
	# GREEN; this step is what failed the job. That is the
	# exact vacuous pass the floor exists for, and it needed
	# no bug to produce.
	- name: Positive control (the run discovered the suite)
	shell: bash
	run: \|
	set -euo pipefail
	log="$RUNNER_TEMP/kit-tests.log"

	# The floor is a collapse detector, not a ratchet. It sits far below
	# the 1397 tests present when this was written so ordinary churn
	# (deleting a suite, folding two together, moving coverage into the
	# app target) never has to touch it, and far enough above zero that a
	# run discovering nothing cannot slip past. If it ever stops
	# discriminating, raise it; do not track the real count with it.
	floor=1000

	line="$(grep -E 'Test run with [0-9]+ tests' "$log" \| tail -1 \|\| true)"
	if [ -z "$line" ]; then
	echo "::error::swift test printed no run summary. Either it executed no tests, or the summary format changed and this control has gone blind. Under both readings the green exit code from the previous step is not evidence that anything was exercised."
	exit 1
	fi

	count="$(printf '%s\n' "$line" \| sed -E 's/.Test run with ([0-9]+) tests./\1/')"
	echo "Reported by the run: $line"

	if [ "$count" -lt "$floor" ]; then
	echo "::error::Only $count tests ran, under the floor of $floor. The suite has collapsed or is being filtered, and a pass over $count tests is not the gate this job exists to be."
	exit 1
	fi
	echo "Positive control OK: $count tests discovered and run, above the floor of $floor."

	# The open question was whether a hosted image can compile this app at all:
	# the app target's deployment floor is macOS 14, but it calls at least one
	# macOS 26 SDK symbol behind an `#available` check
	# (`sharedBackgroundVisibility`), which needs a macOS 26 SDK to compile even
	# though it needs macOS 26 only to RUN. GitHub's images do not always carry
	# the newest Xcode, and a build job that cannot compile the tree is worse
	# than none.
	#
	# Measured on the runner, 2026-08-11: macos-26-arm64 image 20260728.0273 runs
	# macOS 26.5.2 on arm64 with Xcode 26.6 (17F113) selected, Swift 6.3.3, and the
	# macOS 26.5 SDK. Every Xcode from 26.0.1 to 26.6 is installed. That is AHEAD
	# of the dev machine (Xcode 26.4, Swift 6.3), same major version and same
	# architecture, so the job compiles what a developer compiles. It built clean
	# in 49 seconds.
	#
	# What this job is for: the Kit suite cannot see the app target at all.
	# CandelaKit is barred from importing SwiftUI, so every SwiftUI view, every
	# AppKit island and every use of the engine's API from the app side goes
	# uncompiled by `swift test`, and a signature change in the Kit that breaks a
	# call site in Candela/ passes that suite and always will. Compiling the app
	# here is what closes the hole.
	#
	# The job now also RUNS a suite over those sources: CandelaAppTests, a
	# host-free unit-test bundle that compiles the app's own sources into itself
	# and calls their symbols directly, with no TEST_HOST and no launched app.
	# `CandelaMain.main()` never runs inside it, so no status item, no poller and
	# no DDC writer ever exists in the test process; the suite is as safe on a
	# machine with panels attached as it is meaningful on a runner with none.
	#
	# What that covers: view-adjacent derivation (which rows a surface produces,
	# in what order, with what enabled and why), the copy builders, and a render
	# smoke layer asserting a view rasterises at all. What it does NOT cover, and
	# what no green here may be read as covering: display behaviour of any kind,
	# window presentation, reset ordering, `SMAppService` registration, TCC
	# grants. Those need a launched app in front of a person and stay manual.
	app-build:
	needs: scope
	if: needs.scope.outputs.app == 'true'
	# Not `needs: test`. They are independent questions and running them in
	# parallel answers both on every push, where chaining would hide a build
	# break behind an unrelated test failure and cost a second round trip to
	# find it. Neither job is long enough for the sequencing to save anything.
	runs-on: macos-26
	timeout-minutes: 30
	steps:
	- uses: actions/checkout@v4

	- name: Toolchain
	run: \|
	xcodebuild -version
	swift --version

	# Not preinstalled on the image (verified: `which xcodegen` finds nothing).
	# Unpinned, because Homebrew has no ergonomic way to pin a formula version
	# and the generator's output is checked by the build that follows it: if a
	# newer xcodegen emitted a broken project, this job goes red rather than
	# quietly shipping one. The project file itself is generated and
	# gitignored, so there is nothing to compare against anyway.
	# This is the pipeline's only external fetch, so a Homebrew outage is the
	# one failure here that reddens a run for a reason the diff cannot
	# explain. Hence the skip and the retries.
	- name: Install xcodegen
	shell: bash
	run: \|
	set -euo pipefail
	if command -v xcodegen >/dev/null 2>&1; then
	echo "xcodegen already present: $(xcodegen --version)"
	exit 0
	fi
	for attempt in 1 2 3; do
	if brew install xcodegen; then
	exit 0
	fi
	echo "::warning::brew install xcodegen failed (attempt $attempt of 3)"
	sleep $(( attempt * 15 ))
	done
	echo "::error::xcodegen could not be installed after 3 attempts. This is an environment failure, not a result about the diff."
	exit 1

	# project.yml is the source of truth and Candela.xcodeproj is gitignored,
	# so CI has to generate it exactly as a developer does.
	- name: Test contributor build and Release verification tools
	run: python3 -m unittest discover -s tools/build -p 'test_*.py'

	- name: Generate the project
	run: xcodegen generate

	# An explicit -derivedDataPath for the same reason the deploy loop needs
	# one: it puts the product somewhere known, so the check below reads THIS
	# build's output and not whatever a shared location happened to hold.
	# The signing override is deliberate and spelled out rather than defaulted.
	# project.yml signs the app target with the Developer ID identity in both
	# configurations, which a GitHub runner cannot do: it has no keychain and
	# no certificate, and shipping the private key to CI to give it one would
	# be a far worse trade than building unsigned here. Overriding on the
	# command line keeps the failure honest in the other direction too, since
	# a developer whose certificate is missing gets a codesign error instead
	# of a silent ad-hoc fallback that looks like a successful signed build.
	# The entitlements are NOT overridden, so this job still proves the file
	# parses and applies.
	- name: Build the app
	shell: bash
	run: \|
	set -euo pipefail
	xcodebuild -project Candela.xcodeproj -scheme Candela \
	-configuration Debug -derivedDataPath "$RUNNER_TEMP/dd" build \
	CODE_SIGN_IDENTITY="-" DEVELOPMENT_TEAM=""

	# ` BUILD SUCCEEDED ` is a claim about the build, not about a product.
	# Treating it as proof that an app exists is the same mistake as treating
	# a DDC write ACK as proof a monitor changed: check the achieved state. A
	# misconfigured scheme or a target that compiled to nothing both exit 0,
	# so the binary is looked for by name.
	# The digest is logged so the control's evidence outlives the test step
	# below, which writes into this same build directory.
	- name: Positive control (a product actually came out)
	shell: bash
	run: \|
	set -euo pipefail
	app="$RUNNER_TEMP/dd/Build/Products/Debug/Candela.app"
	bin="$app/Contents/MacOS/Candela"
	if [ ! -x "$bin" ]; then
	echo "::error::xcodebuild reported success but produced no executable at $bin. The build is not building what this job claims it is."
	exit 1
	fi
	echo "Positive control OK: $(file -b "$bin")"
	echo "Attested product: $bin"
	echo " sha256: $(shasum -a 256 "$bin" \| cut -d' ' -f1)"
	echo " built: $(date -u -r "$bin" '+%Y-%m-%dT%H:%M:%SZ')"

	# Folded into app-build rather than given a third job, for the same
	# 10x-billing reason the trigger split at the top of this file documents.
	# The test bundle compiles the app's sources, so a separate runner would
	# have to repeat this job's checkout, `brew install xcodegen` and
	# `xcodegen generate` in full before it could run one test, and would buy
	# no signal that running it from here does not already buy. The setup is
	# already paid for; reuse it.
	#
	# Shares the build above's -derivedDataPath. A separate path recompiled
	# every shared dependency a second time inside one job, and the digest
	# logged above is what keeps the control's evidence intact without it.
	#
	# Only the dependencies are recovered. The app's own sources still compile
	# twice, because CandelaAppTests compiles them straight into the test
	# bundle instead of depending on the Candela target, which is how the
	# tests reach internal types with no @testable.
	#
	# Unmeasured as of 2026-08-17, and stated rather than assumed: the render
	# smoke layer drives `ImageRenderer`, which is proven only on an unlocked
	# local session. Whether offscreen rendering works in an xctest process on
	# a headless runner is an open question, and THE FIRST RUN OF THIS STEP IS
	# THE MEASUREMENT. If those tests fail here for environmental reasons, the
	# fix is an environment gate on the smoke layer alone (runs locally, skips
	# on the runner, with the measured fact written back into this comment)
	# while the structural layer keeps running everywhere regardless. The gate
	# is deliberately not built ahead of the evidence: gating on a guess would
	# silently drop coverage that may well work.
	- name: Run the app suite
	shell: bash
	run: \|
	set -euo pipefail
	# pipefail for the same reason the Kit job's "Run the suite" step
	# spells out: a pipeline reports its LAST command's status, so without
	# it this reports tee's, and tee succeeds whatever xcodebuild did.
	# Every failing suite would be published as green.
	xcodebuild test -project Candela.xcodeproj -scheme CandelaAppTests \
	-destination 'platform=macOS' -derivedDataPath "$RUNNER_TEMP/dd" \
	2>&1 \| tee "$RUNNER_TEMP/app-tests.log"

	# The Kit job's control, applied to the app suite for the same reason: a
	# zero exit from a run that discovered nothing is indistinguishable from a
	# pass. A test bundle that built empty, a scheme that stopped listing the
	# target, a filter matching no name: each exits 0 having exercised
	# nothing, and the step above cannot tell those apart from success. So the
	# run's own summary line is read back out of the log, and its absence is
	# itself a failure.
	- name: Positive control (the app suite discovered its tests)
	shell: bash
	run: \|
	set -euo pipefail
	log="$RUNNER_TEMP/app-tests.log"

	# Both halves were demonstrated failing on this runner before this
	# step merged, the same discipline the Kit job applied to itself
	# [MEASURED 2026-08-17]:
	# run 32065655446 one expectation flipped to a wrong value; the
	# job went red at "Run the app suite" itself.
	# run 32065124183 the suite filtered to a name matching nothing.
	# xcodebuild printed a summary of 0 tests and
	# exited 0, so "Run the app suite" stayed GREEN;
	# this step is what failed the job. That is the
	# exact vacuous pass the floor exists for.
	#
	# A collapse detector, not a ratchet, exactly as the Kit job's floor
	# is. It sits far below the count this suite had when the step was
	# written, so ordinary churn (folding two suites together, moving
	# coverage down into the Kit) never has to touch it, and far enough
	# above zero that a run discovering nothing cannot slip past. If it
	# ever stops discriminating, raise it; never track the real count.
	floor=25

	line="$(grep -E 'Test run with [0-9]+ tests' "$log" \| tail -1 \|\| true)"
	if [ -z "$line" ]; then
	echo "::error::The app suite printed no run summary. Either it executed no tests, or the summary format changed and this control has gone blind. Under both readings the green exit code from the previous step is not evidence that anything was exercised."
	exit 1
	fi

	count="$(printf '%s\n' "$line" \| sed -E 's/.Test run with ([0-9]+) tests./\1/')"
	echo "Reported by the run: $line"

	if [ "$count" -lt "$floor" ]; then
	echo "::error::Only $count app tests ran, under the floor of $floor. The suite has collapsed or is being filtered, and a pass over $count tests is not the gate this step exists to be."
	exit 1
	fi
	echo "Positive control OK: $count app tests discovered and run, above the floor of $floor."

	# Release is the configuration that ships, and Debug exercises neither its
	# optimisation level nor the code `#if DEBUG` excludes. The rule about grepping
	# the Release binary for debug markers also had nowhere to live but a person's
	# memory, and a marker has already reached a Release build once.
	#
	# What this job does NOT establish: the runner has no keychain and no
	# certificate, so it builds with the same signing override app-build uses.
	# Signing, the hardened runtime and notarisation stay verified at deploy time.
	release-build:
	needs: scope
	if: needs.scope.outputs.app == 'true'
	runs-on: macos-26
	timeout-minutes: 30
	steps:
	- uses: actions/checkout@v4

	- name: Toolchain
	run: \|
	xcodebuild -version
	swift --version

	- name: Install xcodegen
	shell: bash
	run: \|
	set -euo pipefail
	if command -v xcodegen >/dev/null 2>&1; then
	echo "xcodegen already present: $(xcodegen --version)"
	exit 0
	fi
	for attempt in 1 2 3; do
	if brew install xcodegen; then
	exit 0
	fi
	echo "::warning::brew install xcodegen failed (attempt $attempt of 3)"
	sleep $(( attempt * 15 ))
	done
	echo "::error::xcodegen could not be installed after 3 attempts. This is an environment failure, not a result about the diff."
	exit 1

	- name: Generate the project
	run: xcodegen generate

	# Same signing override as the Debug job, for the same reason: no keychain
	# on a hosted runner. The entitlements are NOT overridden, so this build
	# still proves the file parses and applies under Release.
	- name: Build the app (Release)
	shell: bash
	run: \|
	set -euo pipefail
	xcodebuild -project Candela.xcodeproj -scheme Candela \
	-configuration Release -derivedDataPath "$RUNNER_TEMP/dd-release" build \
	CODE_SIGN_IDENTITY="-" DEVELOPMENT_TEAM=""

	- name: Debug markers are absent from the Release build
	run: tools/build/check-release-markers.sh "$RUNNER_TEMP/dd-release/Build/Products/Release/Candela.app"

	verification:
	name: App verification
	if: ${{ always() }}
	needs: [scope, test, app-build, release-build]
	runs-on: ubuntu-latest
	timeout-minutes: 5
	steps:
	- uses: actions/checkout@v4
	- name: Require every selected app job to pass
	env:
	NEEDS_JSON: ${{ toJSON(needs) }}
	run: python3 tools/ci/verification.py gate app

Navigation Menu

Repository navigation

Add firmware trace and direct C comparison to OLED evidence #159

Workflow file

Add firmware trace and direct C comparison to OLED evidence #159

Uh oh!

Workflow file for this run