ci: run integration tests as nix flake checks - #3342
Draft
kradalby wants to merge 10 commits into
Draft
Conversation
kradalby
force-pushed
the
integration-nix-checks
branch
2 times, most recently
from
June 23, 2026 19:50
a227589 to
78aeef9
Compare
kradalby
commented
Jun 24, 2026
kradalby
force-pushed
the
integration-nix-checks
branch
4 times, most recently
from
June 25, 2026 08:08
36c5a61 to
4a3e731
Compare
The hermetic nix checks cannot pull tailscale images at eval time, so the tested version set and the per-arch image digests are pinned in integration/tailscale-versions.json. capver.IntegrationVersions() is the single source the suite, the drift guard and the pin generator share; the generator (tools/tailscale-versions, run via go generate) refreshes it after a capver bump and reuses already-present pins so a routine generate makes no network call.
Replaces the Dockerfile builds for the flake-check path. headscale is built from the Go-filtered source; one tailscale-HEAD image serves client, daemon, derper and the websocket-DERP test. Released tailscale versions and postgres are pinned registry pulls, arch following the build system. Tools the suite used to apk-add (curl/ssh/dig/python3) and the SSH login users are baked in at build time so no container is mutated at startup, and busybox su is included so taildrive can fork its per-user fileserver.
Read the HEADSCALE_INTEGRATION_*_IMAGE knobs through one envknob helper and assert required tooling is present instead of installing it at runtime (the nix VM is offline). dsic waits for both the derper cert and key before starting derper, so the container can't die between the two uploads. The nested-docker VM is slower than a host daemon, so every integration timeout (convergence, docker exec, peer sync, pool wait) now flows through one dockertestutil.ScaleTimeout honouring HEADSCALE_INTEGRATION_TIMEOUT_SCALE.
Each test becomes a hermetic check: a VM boots docker, loads the prebuilt images and runs that one test (nix/tests/integration.nix), so the suite runs offline as nix build .#checks.<name>. The generator emits the test list as nix: heavy multi-subtest tests are split into per-subtest checks, and tests the nested docker VM can't host are excluded for the legacy pipeline. flake.nix wires the checks, the images, and the Go build/test/lint checks (with the version JSON on the embed whitelist).
…cy pipeline CI runs nix build .#checks.<system>.integration-* instead of the bespoke docker matrix — the generated checks are the matrix. The tests the hermetic VM can't run (real-internet DERP, docker cable-pull) run via cmd/hi in a slim legacy workflow, which also keeps cmd/hi and the integration Dockerfiles exercised. check-tests guards the pinned versions against capver drift.
Garnix builds every flake check and package directly, so the GitHub
Actions wrappers around them are redundant:
- nix-checks checks.{build,gotest,golangci-lint,formatting}
- nix-module-test checks.headscale (NixOS module test)
- test-integration the integration-* NixOS-VM check matrix
- build packages.headscale (nix build)
Kept are the pipelines garnix cannot reproduce: servertest (excluded
from the flake), integration-legacy (needs real DERP, so it cannot run
hermetically), and the check-generated / check-tests generators that
keep the nix integration matrix and protobuf output in sync.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxukxKem3HU4i5CqNrvvdj
kradalby
force-pushed
the
integration-nix-checks
branch
from
July 1, 2026 12:15
89b4898 to
a4a3633
Compare
Rebasing onto main pulled in TestK8sOperator and the OAuth client CLI tests; regenerate integration/tests.nix so the NixOS-VM check matrix runs them too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WxukxKem3HU4i5CqNrvvdj
kradalby
force-pushed
the
integration-nix-checks
branch
from
July 1, 2026 13:57
a4a3633 to
566a978
Compare
The self-hosted garnix builder runs many checks at once (nix max-jobs is 16 on our box), so the 8 GB integration VMs overcommit its 64 GB RAM (16 * 8 = 128 GB) and the whole matrix thrashes with none completing. Drop each VM to 3.5 GB (16 * 3.5 = 56 GB, leaving host headroom) so the checks actually run; raise it back toward 8 GB once the builder caps max-jobs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WxukxKem3HU4i5CqNrvvdj
The 3.5 GB reduction was a diagnostic. The matrix wedged because the self-hosted builder runs unbounded max-jobs and does not cancel superseded runs (stacked ghost VM builds), not because 8 GB VMs are too big. Keep the tested-safe size; the fix is capping the builder's max-jobs so memorySize * max-jobs fits its RAM. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WxukxKem3HU4i5CqNrvvdj
Running each of the 161 integration tests as its own 8 GB NixOS-VM check overcommits a single self-hosted garnix builder: garnix fans them out past the builder's RAM and the OOM killer takes down the build (and the garnix coordinator VM itself). Group the matrix into 6 sqlite batches + 1 postgres batch, each running its tests sequentially (-test.parallel=1) in one 6 GB VM, so at most 7 VMs (~42 GB) ever run regardless of garnix's build concurrency. Trade-off: per-test checks collapse to per-batch (a failure shows in the batch log, not as its own check). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WxukxKem3HU4i5CqNrvvdj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
hypothetical pr to see if we can speed up and run the CI tests inside a Nix enclosure. The goal is to have to rebuild less things by using Nix caches for things that dont change, and to make them run on Garnix which I hope to set up.