Tests don't run in CI on macOS, Windows or Linux, and the CPU/GPU equivalence tests never run in CI (gap 9)

Summary

cargo test runs in CI only on FreeBSD. Nothing runs the tests on macOS, Windows or Linux x64, the three platforms people install LightCraft on, so platform-specific breakage reaches main and stays there until someone builds on that platform by hand. The CPU/GPU equivalence tests, the guard that keeps the GPU twin matching the CPU reference, never run in CI on any platform: they skip when there is no GPU adapter, and the code refuses the software adapters CI runners have.

This is gap 9 in docs/gaps.md (beta blocker, 6–12 h). This issue proposes a fix that stays inside the current design and adds no new runner types.

Evidence

.github/workflows/ has freebsd.yml (the only workflow that runs cargo test), packaging-lint.yml, release.yml and windows-arm64.yml. release.yml builds packages on macOS, Windows and Linux but runs no tests. windows-arm64.yml runs the CLI natively and has no cargo test job (its header comment explains why).
main does not build or pass its tests on macOS (995d5ea4): app won't compile, 12 tests fail, one clippy lint #989: on 2026-10-11, main did not compile on macOS (eframe::App::on_exit called with one argument in the macOS-only quit path), and 12 tests failed there. Nothing in CI could have caught it. The follow-ups (Issue #989: the lock test finds true on the PATH #990, Issue #989: clippy 1.99, fill_box reads its buffer with as_chunks::<3>() (item 4) #997, Issue #989: odd_folders_never_fail_the_check skips non-UTF-8 names on APFS (item 3, one of eleven) #1003, Issue #989: the ten sidecar repair tests also pass on a case-insensitive volume (item 3, ten of eleven) #1007, Issue #989: the Export dialog height test expects the macOS row (Add to Apple Photos, 876 pt) #1018) are still open.
crates/gpu/tests/equivalence.rs:19-25: gpu() prints skipped: no GPU adapter and the test passes. fallback.rs, memory.rs, shutdown.rs and shutdown_first.rs do the same.
crates/gpu/src/ctx.rs:299-302: Gpu::new returns Err for wgpu::DeviceType::Cpu ("the CPU pipeline is faster"). That is the right call for the app. The side effect is that lavapipe (Linux) and WARP (Windows), the only adapters on GitHub's standard runners, are never used, so the equivalence suite has never run in CI.

What a test run on the three platforms finds today

I ran the workflow below on my fork, on main at 76b5377. All runners were standard GitHub-hosted ones with a cold cache.

	`main` alone	`main` + #990, #997, #1003, #1007, #1018
Linux x86_64, `cargo xtask ci`	fails at clippy (`clippy::chunks_exact_to_as_chunks`, rustc 1.99, #989 item 4)	all 8 steps pass (39 min cold)
macOS arm64, clippy + test	clippy as above; `lightcraft` does not compile (#989 item 1)	pass (clippy 3 min, tests 19 min)
Windows x86_64, clippy + test	clippy as above	2 new Windows-only failures, below
GPU tests on a software adapter / Metal	lavapipe: 16/16 equivalence tests pass	lavapipe, WARP and Metal all pass

Two Windows failures that nobody had reported:

clippy, Windows only: method descriptor_copy is never used at crates/catalog/src/lock.rs:140 (-D dead-code). The method is #[cfg(test)], and its only caller is a #[cfg(unix)] test (crates/catalog/src/tests_lock.rs:114-119).
editors::tests::desktop_entries_give_their_program (apps/lightcraft/src/editors.rs:222) fails on Windows: left: None, right: Some("/home/simon/Apps/photocraft-1.2-linux-x86_64.AppImage"). The .desktop parser presumably checks for an absolute path, and /home/... is not absolute on Windows. That run stopped at the first failing test binary, so there may be more. The workflow now uses --no-fail-fast on macOS and Windows so every failure shows.

These two are not fixed here. They belong in their own PRs, the same way the #989 items were split.

The equivalence suite on software adapters

With the opt-in described below, all 134 compared renders (16 tests) stay inside the documented bounds (mean |Δ| < 0.5 LSB, max ≤ 3 LSB). The results match the Apple M4 Pro numbers in docs/gpu-pipeline.md, and no tolerance was changed:

adapter	renders	max \|Δ\|	worst mean \|Δ\|	equivalence time
lavapipe (Mesa, LLVM 20), local	134	1 LSB (110 renders), 0 (24)	0.0003	19 s
lavapipe, `ubuntu-latest`	134	1 LSB	0.0003	50 s
WARP (DX12, FXC), `windows-latest`	134	1 LSB (109), 0 (25)	0.0003	446 s
Metal (paravirtual GPU), `macos-latest`	16 tests pass	(not printed)		29 s

One test assumed more than a WebGPU-sized device. windows_of_a_big_photo_share_one_uploaded_source failed on lavapipe (left: 0, right: 1 uploads). lavapipe's max_storage_buffer_binding_size is 128 MiB, which is the WebGPU default. The test's 24 MP source is 288 MB, so by design (render.rs:619) geometry ran on the CPU and nothing was uploaded. This is not a drift between CPU and GPU. Its sibling, a_window_does_not_keep_the_whole_source_on_the_device, passed on lavapipe without checking anything, for the same reason.

Proposed fix (within the current design)

Test-only opt-in for software adapters. Add LIGHTCRAFT_GPU_ALLOW_SOFTWARE=1, read in Gpu::new. The app never sets it, and the default behaviour does not change: software adapters are still refused. This follows LIGHTCRAFT_GPU_LIMITS, which also exists to reproduce an adapter. A unit test covers the decision (refuse_adapter).
Size the two big-source tests to fit a 128 MiB binding. Use 3200 × 3200 (117 MiB): above the 96 MiB RETAIN_SOURCE_BYTES, so the source is still shared, and within the WebGPU default, so it is uploaded on every adapter. On a real GPU they test the same code path. The first test still catches the Missing a native full resolution preview of pictures #323 regression: a 117 MiB source kept in the stages is more than its 100 MiB bound.
.github/workflows/ci.yml, standard runners only, on every pull request and every push to main, with concurrency cancel-in-progress:
- Linux x86_64 (ubuntu-latest): cargo xtask ci, the same gate contributors run locally, so new xtask steps are checked automatically. Then the GPU test targets with lavapipe (mesa-vulkan-drivers).
- macOS arm64 (macos-latest) and Windows x86_64 (windows-latest): clippy and cargo test --workspace --no-fail-fast. These are the steps whose result depends on the platform: cfg(target_os) code and file-system behaviour. fmt, parity, layers, assets and wasm give the same answer everywhere, so they run once, on Linux. Then the GPU test targets on Metal and WARP.
- The GPU step names the test targets with --workspace (--test equivalence --test fallback ...) so it reuses the workspace build, and fails if anything still prints skipped: no GPU adapter. Without that check, a runner-image change could silently turn the guard off again.
- Swatinem/rust-cache (the pin release.yml already uses) caches dependencies only, and saves only from main (save-if). Pull requests restore from it, so the 10 GB repository cache holds one entry per job.
- It uses only the actions and pins already in release.yml (checkout, dtolnay/rust-toolchain, rust-cache). No new third-party actions.
- xtask already sizes build jobs and test threads by available RAM (ci_jobs). The macOS runner (3 cores, 7 GB) passed with the defaults.
Docs: gap 9 and the CI mentions in ROADMAP.md / docs/target-app-parity.md, plus the new variable in docs/gpu-pipeline.md.

Cost: standard GitHub-hosted runners are free for public repositories. Measured cold-cache durations: Linux 41 min, Windows 28 min, macOS 23 min, all in parallel. Warm runs should be shorter, because dependencies are restored from main's cache.

For the maintainers to decide (not in the PR)

Per-PR or not. CI: FreeBSD and Windows ARM64 nightly and on release, not per PR #901 moves FreeBSD and Windows ARM64 to nightly because PR CI waits on runners shared by ~14 craft apps. This workflow uses the same org-wide pool: 1 macOS and 2 standard jobs per PR. If that is too much, two lines change it: run macOS and Windows on push to main plus a nightly schedule, and keep Linux per-PR. main does not build or pass its tests on macOS (995d5ea4): app won't compile, 12 tests fail, one clippy lint #989 suggests that macOS per-PR is what would have caught it.
WARP time. The equivalence suite takes ~7.5 min on WARP. If that is too slow per PR, run the Windows GPU step only on main and keep lavapipe per-PR.
Toolchain. The workflow uses stable, like release.yml, so a new clippy lint can turn main red overnight (main does not build or pass its tests on macOS (995d5ea4): app won't compile, 12 tests fail, one clippy lint #989 item 4). A rust-toolchain.toml pin, bumped deliberately, would avoid that. That is a project-wide decision, so it is not part of this proposal.
Required checks. Once main is green on all three (after the main does not build or pass its tests on macOS (995d5ea4): app won't compile, 12 tests fail, one clippy lint #989 PRs and the two Windows fixes), the three jobs could become required status checks.
Relation to ci: add Linux x64 CI workflow (partially closes gap #9) #941 (Linux-only CI, open, with review comments). This proposal covers its Linux job and addresses those comments: no retry: inputs, no /usr/lib cache, no path filter, trailing newline. It also adds the wasm target, which that run lacked (ci failed at wasm), plus macOS, Windows and the GPU guard. Both add .github/workflows/ci.yml, so only one of them should land.

Test plan

cargo test -p lightcraft-gpu, default environment: the GPU tests skip as before, and software_adapters_need_the_opt_in passes.
LIGHTCRAFT_GPU_ALLOW_SOFTWARE=1 VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.json cargo test -p lightcraft-gpu -- --nocapture: all 16 equivalence tests and the fallback, memory and shutdown tests run on llvmpipe and pass.
The workflow, run on a fork: the Linux, macOS and Windows jobs and their GPU steps. Results are in the table above.
actionlint reports nothing for the workflow file.

test(gpu): storytold#1050 opt-in software adapter for the equivalence…

ci: storytold#1050 run the tests on Linux, macOS and Windows

docs: storytold#1050 gap 9 shrinks: tests in CI on the three desktop …

ci: storytold#1050 run the tests on Linux, macOS and Windows

docs: storytold#1050 gap 9 shrinks: tests in CI on the three desktop …

Update: the full Windows picture, and where each failure is being fixed

The first Windows run stopped at the first failing test binary. The workflow now runs cargo test --workspace --no-fail-fast on macOS and Windows. A second fork run (main at 76b5377 + the workflow + #990, #997, #1003, #1007, #1018) shows everything that fails there:

job	result
Linux x86_64, `cargo xtask ci` + GPU tests on lavapipe	pass (all 8 steps; 16/16 equivalence tests)
macOS arm64, clippy + tests + GPU tests on Metal	pass
Windows x86_64, clippy	fails: `dead_code` on `LibraryLock::descriptor_copy` (`crates/catalog/src/lock.rs:140`)
Windows x86_64, tests	22 failures in 2 binaries (below)
Windows x86_64, GPU tests on WARP	pass (134 renders, max 1 LSB, mean ≤ 0.0003)

Windows test failures, all on main before this change (the PR touches no engine or app code):

lightcraft (bin): editors::tests::desktop_entries_give_their_program (apps/lightcraft/src/editors.rs:222, left: None). A /home/... Exec path is not absolute to Path::is_absolute on Windows.
lightcraft-engine (lib), 21 tests: 762 passed and 21 failed. They fall into two groups:
- Path separators in fixtures. The tests build paths with / and the code joins with \. For example, rename.rs:1314 expected ...\Archive/Trip\b.jpg and got ...\Archive\Trip\b.jpg; tests_filesystem_history.rs:346 expected /p/Trip.jpg and got /p\Trip.jpg; tests_folders.rs:624 expected ...\out/a.jpg. The others in this group are merged_rename_failure_keeps_rolled_back_photo_identity and merged_renames_coalesce_physical_moves_and_keep_retry_inverses_and_journal.
- Case-insensitive NTFS. The sidecar::tests_repair_companions (3), tests_repair_provenance (7) and tests_repair_saves (2) tests fail, plus 4 tests_filesystem_history retries. The injected failure never fires (left: 4, right: 1 sidecars moved; unwrap_err() on Ok("Rename 1 Photo")). This is the Rename Photos: on a case-insensitive volume the sidecar a.xmp becomes Trip.XMP #1006 / Issue #989: the ten sidecar repair tests also pass on a case-insensitive volume (item 3, ten of eleven) #1007 mechanism. Issue #989: the ten sidecar repair tests also pass on a case-insensitive volume (item 3, ten of eleven) #1007's test-side fix covers APFS but not NTFS: Windows still fails with Issue #989: the ten sidecar repair tests also pass on a case-insensitive volume (item 3, ten of eleven) #1007 merged.

#1052 (OneSlips, open) already fixes all of these: the clippy lint, the editors test, the case-insensitive rename / save bugs (#1006) and the separator fixtures. So nothing new needs filing. I confirmed this with a third fork run that also had #1052 merged on top (run 38174552427). All three jobs are green:

job	clippy	tests	GPU step
Linux x86_64	`cargo xtask ci`: all 8 steps, 39 min	(in xtask ci)	lavapipe: 134 renders, max 1 LSB, mean ≤ 0.0003, suite 46 s; step 2 min
macOS arm64	pass, 4 min	pass, 26 min	Metal (paravirtual): 134 renders, max 2 LSB in one case (point colour range overlay, 0.0002% of pixels over 1), the rest ≤ 1, mean ≤ 0.0003, suite 29 s
Windows x86_64	pass, 5 min	pass, 33 min	WARP: 134 renders, max 1 LSB, mean ≤ 0.0003, suite 436 s (111 s in an earlier run)

cargo xtask ci on the same merge also passes all 8 steps locally (Linux, rustc 1.99). #1052 and #997 both change fill_box in crates/pipeline/src/spots.rs, and #1052 and #1007 change the same assertion in tests_repair_saves.rs, so whichever lands second needs a small rebase.

Change to the workflow since the issue

The Linux GPU step now runs cargo test -p lightcraft-gpu --tests. Its earlier --workspace --test … form rebuilt about 190 crates after cargo xtask ci (7 min); this form takes about 2 min. macOS and Windows keep the --workspace form, which reuses the test step's build (2–8 s of building).

Prior art since the issue

#1083 and #1084 (a nightly macos.yml) were opened and then withdrawn by their author in favour of the open #989 PRs and #901's nightly direction. Per-PR or nightly remains the maintainers' call (see "For the maintainers to decide" in the issue body). The PR keeps the platform jobs per-PR and explains how to move them.

test(gpu): storytold#1050 opt-in software adapter for the equivalence…

ci: storytold#1050 run the tests on Linux, macOS and Windows

docs: storytold#1050 gap 9 shrinks: tests in CI on the three desktop …

ci: storytold#1050 run the tests on Linux, macOS and Windows

docs: storytold#1050 gap 9 shrinks: tests in CI on the three desktop …

fix: WindowsのCI失敗修正を統合 storytold#1050

ci: 3OSとGPUのCIを既存PRへ統合 storytold#1050

Sorry, I missed this issue and the existing CI PRs (#1148, #1159, #941) before opening #2005, which duplicates them. I'll close #2005 in favour of whichever one you pick.

What's still useful from it: #2004 fixes 3 tests that fail on windows-latest (path separators in the folder-drop and export tests, and a missing wait in the offline test). Any of these CI PRs will hit those failures on Windows.
Green run on all 3 OS for reference: https://github.com/francoisgn/lightcraft/actions/runs/38873633880

fix: storytold#1050 clippy on macOS and Windows: one Quit / About / S…

docs: storytold#1050 gap 9 closed: tests in CI on macOS, Windows and …

test(gpu): storytold#1050 opt-in software adapter for the equivalence…

ci: storytold#1050 run the gate on macOS and Windows, and the GPU tes…

fix: storytold#1050 clippy on macOS and Windows: one Quit / About / S…

docs: storytold#1050 gap 9 closed: tests in CI on macOS, Windows and …

test(gpu): storytold#1050 opt-in software adapter for the equivalence…

fix: storytold#1050 clippy on macOS and Windows: one Quit / About / S…

docs(gpu): storytold#1050 run the GPU tests locally on a software ada…

docs(gpu): storytold#1050 the skip-log check catches every GPU skip a…

test(gpu): storytold#1050 opt-in software adapter for the equivalence…

docs(gpu): storytold#1050 run the GPU tests locally on a software ada…

docs(gpu): storytold#1050 the skip-log check catches every GPU skip a…

docs(gpu): storytold#1050 the skip check also expects the device-memo…

One more gap to fold in here. The Linux gate runs the tests now, but it has no corpus. This run on main reports corpus: 0 of 462 pinned raws, 0 images, 0 PngSuite files, so every corpus-backed test skips and passes. That includes the per-model checks behind the verified-models count in ROADMAP.md, which CI therefore never exercises.

A nightly job that runs cargo xtask corpus --download before the tests would cover it without adding time to pull requests.

Agreed. I ran it locally to see what the gap is hiding. With the full pinned corpus (LIGHTCRAFT_CORPUS set, all 460 raws plus the D750 set), lightcraft-raw, -codecs and -engine on main (8ef35987d) had 12 failures, all green in CI because their tests skip without the corpus:

CR3: probe_info's baseline_exposure differs from decode since #3805, failing 10 corpus tests #4158: since Extend CR3 white level to actual sensor saturation #3805 (Nov 5), a full CR3 decode sets a pixel-derived baseline_exposure that probe_info can't, so 10 CR3 tests fail the probe-equals-decode check. corpus_raw_decodes stops at the first failing file, so the rest of the 460-file sweep isn't checked until that's fixed.
corpus_raf_colour_and_white_balance still expects relative white balance for the X-T2 and X-T20 after #730 #4159: Kelvin white balance for raws whose camera has no colour matrices #730 (Oct 21) gave the X-T2 / X-T20 Kelvin white balance, but corpus_raf_colour_and_white_balance still expects them relative.
corpus_rw2_distortion_matches_the_camera_jpeg fails on the D-Lux 7 since #3509's 8-bit dither #4160: pipeline: apply 8x8 ordered Bayer dithering to prevent display banding #3509's 8-bit dither (Nov 5) lowers the D-Lux 7 corner correlation in corpus_rw2_distortion_matches_the_camera_jpeg. Bisected; it passes with dithering off.

So corpus-only breakage is already landing within days.

Size, which decides how a nightly can work:

The pinned corpus is 14.1 GB in 462 files, median 24.5 MB. It's essentially one file per model or format variant, so no small subset still covers every verified model.
Files up to 30 MB: 307 files, 5.6 GB. Up to 60 MB: 421 files, 10.1 GB.
The Actions cache is 10 GB per repository by default and is shared with the cargo caches, so the full corpus can't be cached there.
Downloading 14 GB from raw.pixls.us every night isn't fair to a community-run site, so a nightly needs a cache or a mirror either way.
Run time is modest: those three crates take 9.5 min in the dev profile on a 32-core machine.

Options, roughly in order:

A self-hosted runner that keeps the corpus on disk, for a scheduled job only. Full coverage, nothing downloaded after the first run. An infrastructure and security call for maintainers.
A cached subset nightly (e.g. the ≤30 MB files, 5.6 GB) plus the full set weekly or on demand. Partial nightly coverage, and the cache competes with the cargo caches.
A pinned mirror like storytold/photocraft-corpus, which takes the load off raw.pixls.us but still downloads 14 GB per run.

Whichever is chosen, the job would call the shared gate (it already has command and free-disk-space), and any corpus caching support belongs in craft-common-actions, per the CI rule. I can open that issue there once there's a preferred option.

Navigation Menu

Repository navigation

Tests don't run in CI on macOS, Windows or Linux, and the CPU/GPU equivalence tests never run in CI (gap 9) #1050

Summary

Evidence

What a test run on the three platforms finds today

The equivalence suite on software adapters

Proposed fix (within the current design)

For the maintainers to decide (not in the PR)

Test plan

Update: the full Windows picture, and where each failure is being fixed

Change to the workflow since the issue

Prior art since the issue

6 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Tests don't run in CI on macOS, Windows or Linux, and the CPU/GPU equivalence tests never run in CI (gap 9) #1050

Description

Summary

Evidence

What a test run on the three platforms finds today

The equivalence suite on software adapters

Proposed fix (within the current design)

For the maintainers to decide (not in the PR)

Test plan

Activity

THEtheChad commented on Oct 11, 2026

Update: the full Windows picture, and where each failure is being fixed

Change to the workflow since the issue

Prior art since the issue

francoisgn commented on Oct 16, 2026

6 remaining items

AlexChambers commented on Nov 9, 2026

THEtheChad commented on Nov 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions