Skip to content

Commit f7739c4

Browse files
hamishmackangerman
andauthored
Update haskell.nix and use nixpkgs-2511 (#220)
* Update to nixpkgs-2511 and bump dependencies - Switch from nixpkgs-2411 to nixpkgs-2511 - Update cabal-experimental to stable-haskell/feature/cross-compile - Bump haskell.nix, head.hackage, HLS 2.10→2.11 - Update cabal build configuration to inline constraints - Fix flake packages attribute naming (dots → dashes) * Enable all platforms and remove ghc810 - Add aarch64-linux to supportedSystems - Re-enable macOS and aarch64-linux in CI workflows - Remove ghc810 from compiler list - Update lints workflow to use ghc96 * Replace recursive-nix -env jobs with devShellTools The old -env job generation used recursive-nix to run `nix print-dev-env` inside a derivation, which is not supported on remote builders. Replace with pure evaluation-time approach: - Use devShellTools.unstructuredDerivationInputEnv to extract environment variables from mkShell derivations - Construct PATH via lib.makeBinPath from flattened buildInputs - Filter internal nix variables, keeping only user-defined ones - Generate self-contained wrapper scripts at eval time - Update ghcr-upload.sh to match new -env.sh naming * Fix nixpkgs-2511 build failures: happy, openssl, postgresql - happy: disable disallowGhcReference check — happy-2.1.7 transitively references GHC through happy-lib - openssl musl: skip flaky OCSP test 82-test_ocsp_cert_chain.t - postgresql musl: comprehensive fix for pkgsCross.musl64 which doesn't set isStatic=true, causing all optional features to default on: - Disable jitSupport, perlSupport, pythonSupport, tclSupport - Override llvmPackages_20 to prevent LLVM stdenv switch - Disable LTO (GCC + GNU ld .ltrans failures) - Clear outputChecks, disallowedReferences, separateDebugInfo - Break dev↔out and lib↔out reference cycles with remove-references-to * Add -env-test Hydra jobs to validate devcontainer environments Source each -env.sh script in a sandbox and verify that ghc, cabal, and pkg-config are functional. Optionally checks HLS for non-minimal, non-JS, non-Windows shells with compiler < 9.11. Catches PATH construction errors, missing packages, and broken shellHooks that would produce unusable containers. Not yet in `required` aggregate. * Fix env wrapper with stdenv/setup, gate GHCR uploads to main The devShellTools approach only captured Nix-level derivation attributes, missing hook-computed variables (NIX_CFLAGS_COMPILE, NIX_LDFLAGS, PKG_CONFIG_PATH, etc.) that stdenv setup hooks produce at shell init time. This caused downstream "Missing C library" errors. Fix: export all drvAttrs (including stdenv, buildInputs, initialPath) then source $stdenv/setup at runtime, exactly like `nix develop` does. This runs cc-wrapper, pkg-config-wrapper, and all other setup hooks. Also fix env-tests to save $out before sourcing (setup.sh resets it), gate GHCR uploads to main branch (prevents PR pushes from overwriting production images), gate hello.yml to main, and add pr-validate.yml for PR closure validation via Hydra cache. * Fix PR validation: paginated API response and devx invocation gh api --paginate returns multiple JSON objects (one per page). The old jq heredoc processed each page separately, producing multiple arrays (some empty []) which GitHub Actions rejected as invalid output format. Use jq --slurp to merge all pages into a single array before filtering. Also fix the validate step: the devx wrapper sources $1 as a file path, it doesn't support bash-style -c. Write smoke test commands to a temp file instead. * Fix devx wrapper: set $out for stdenv/setup outside Nix builds stdenv's setup.sh calls _assignFirst which requires $out to be set for output variable assignment. Inside a Nix build the builder sets $out automatically, but when running the devx wrapper directly (containers, CI validation) $out is unset and setup.sh fails with: error: _assignFirst: could not find a non-empty variable whose name to assign to outputDev. Fix in two places: - mkEnvScript: wrapper sets $out to a temp dir when unset - pr-validate.yml: set $out before invoking cached wrappers that don't yet include the mkEnvScript fix * Filter discover to only successfully built check-runs Hydra posts check-runs at evaluation time before builds complete. When flake.nix changes cause new derivation hashes, the discover step would pick up store paths not yet available in any cache, causing nix-store -r to fail on GH runners. Filter for conclusion=="success" and valid /nix/store/ paths to only validate closures that are actually built and cached. * Set all Nix builder runtime vars for devx wrapper outside builds setup.sh runs with set -eu and expects NIX_BUILD_TOP, TMPDIR, out, and other variables that the Nix builder sets at runtime. The previous fix only set $out; NIX_BUILD_TOP was the next failure. Set all required builder runtime variables (NIX_BUILD_TOP, TMPDIR, TMP, TEMP, TEMPDIR, NIX_STORE, out) in both mkEnvScript and the CI validate step. Verified locally on hydra: both static (ghc96-static-env) and dynamic (ghc98-minimal-env) wrappers work correctly. --------- Co-authored-by: Moritz Angermann <moritz.angermann@gmail.com>
1 parent b153ade commit f7739c4

File tree

10 files changed

+493
-129
lines changed

10 files changed

+493
-129
lines changed

.github/workflows/aarch64-linux.yml

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,7 @@
11
name: DevX closures for aarch64-linux
22

33
on:
4-
# Disabled until we have a new aarch64-linux hydra builder
5-
# push:
4+
push:
65

76
jobs:
87
build:

.github/workflows/hello.yml

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ name: Test that we can build the hello world example form hackage
22

33
on:
44
push:
5+
branches: [main]
56
workflow_dispatch:
67

78
jobs:
@@ -11,13 +12,11 @@ jobs:
1112
fail-fast: false
1213
matrix:
1314
platform:
14-
- x86_64-darwin
1515
- x86_64-linux
16-
# Skipping because we do not have runners for these set up.
17-
#- aarch64-darwin
18-
#- aarch64-linux
16+
- x86_64-darwin
17+
- aarch64-darwin
18+
- aarch64-linux
1919
compiler-nix-name:
20-
- ghc810
2120
- ghc96
2221
- ghc98
2322
- ghc910
@@ -34,9 +33,6 @@ jobs:
3433
- false
3534
- true
3635
exclude:
37-
# Just cross compiling javascript with ghc 9.6 and above
38-
- compiler-nix-name: ghc810
39-
target-platform: "-js"
4036
# Windows cross compilation only works on x86_64 right now.
4137
- platform: aarch64-darwin
4238
target-platform: "-windows"

.github/workflows/lints.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ jobs:
4242
nix_path: nixpkgs=channel:nixos-unstable
4343
- run: |
4444
nix profile install github:Kha/nixprof
45-
nixprof record nix develop .#ghc8107 --accept-flake-config
45+
nixprof record nix develop .#ghc96 --accept-flake-config
4646
nixprof report -p
4747
nixprof report -a
4848
nixprof report -s

.github/workflows/main.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,10 @@ jobs:
4545
echo "matrix=$MATRIX" >> $GITHUB_OUTPUT
4646
4747
# We need this process step in here, because we have in excess of 256 jobs.
48+
# Only upload to GHCR from the main branch to prevent PR pushes from
49+
# overwriting production container images.
4850
process:
51+
if: github.ref == 'refs/heads/main'
4952
needs: discover
5053
runs-on: ubuntu-latest
5154
strategy:

.github/workflows/pr-validate.yml

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
name: Validate DevX containers (PR)
2+
3+
# Validate devcontainer closures on pull requests by fetching
4+
# Hydra-cached build outputs and smoke-testing them. This replaces
5+
# the GHCR upload pipeline for PRs — actual uploads only happen
6+
# from the main branch (gated in main.yml).
7+
on:
8+
pull_request:
9+
10+
env:
11+
GH_TOKEN: ${{ github.token }}
12+
13+
jobs:
14+
wait-for-hydra:
15+
name: Wait for Hydra status
16+
runs-on: ubuntu-latest
17+
steps:
18+
- uses: input-output-hk/actions/wait-for-hydra@latest
19+
with:
20+
check: required
21+
22+
discover:
23+
needs: wait-for-hydra
24+
name: Discover -env closures
25+
runs-on: ubuntu-latest
26+
outputs:
27+
matrix: ${{ steps.set-matrix.outputs.matrix }}
28+
steps:
29+
- name: Discover Hydra -env check-runs
30+
id: set-matrix
31+
run: |
32+
# Fetch all check-runs for this commit from Hydra.
33+
# --paginate returns multiple JSON objects (one per page),
34+
# so we pipe through jq --slurp to merge them before filtering.
35+
RUNS=$(gh api "repos/$GITHUB_REPOSITORY/commits/${{ github.event.pull_request.head.sha }}/check-runs" --paginate)
36+
37+
# Pick a representative subset: one -env per platform to keep
38+
# the matrix small (avoids 80+ jobs on every PR push).
39+
# Only include successfully completed builds — Hydra posts
40+
# check-runs at eval time before builds finish, so in-progress
41+
# or failed builds would have store paths not yet in caches.
42+
FILTERED=$(echo "$RUNS" | jq -s -c '
43+
[.[].check_runs[]
44+
| select(.name | endswith("-env"))
45+
| select(.conclusion == "success")
46+
| select(.output.summary | . != null and startswith("/nix/store/"))
47+
| { "config": .name,
48+
"build_path": .output.summary,
49+
"short_name": (.name | sub("^[^.]+\\."; "")) }]
50+
| group_by(.short_name)
51+
| map(.[0])
52+
| sort_by(.config)
53+
| .[0:8]
54+
')
55+
56+
echo "Selected closures for validation:"
57+
echo "$FILTERED" | jq .
58+
echo "matrix=$FILTERED" >> $GITHUB_OUTPUT
59+
60+
validate:
61+
needs: discover
62+
if: needs.discover.outputs.matrix != '[]' && needs.discover.outputs.matrix != ''
63+
runs-on: ubuntu-latest
64+
strategy:
65+
fail-fast: false
66+
matrix:
67+
job: ${{ fromJson(needs.discover.outputs.matrix) }}
68+
name: Validate ${{ matrix.job.short_name }}
69+
steps:
70+
- name: Install Nix
71+
uses: cachix/install-nix-action@v31
72+
with:
73+
extra_nix_config: |
74+
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= loony-tools:pr9m4BkM/5/eSTZlkQyRt57Jz7OMBxNSUiMC4FkcNfk=
75+
substituters = https://cache.iog.io/ https://cache.zw3rk.com/ https://cache.nixos.org/
76+
77+
- name: Validate closure
78+
env:
79+
SHELL_NIX_PATH: ${{ matrix.job.build_path }}
80+
run: |
81+
set -euo pipefail
82+
83+
echo "::group::Realize closure from Hydra cache"
84+
nix-store -r "$SHELL_NIX_PATH"
85+
echo "::endgroup::"
86+
87+
CLOSURE_PATHS=$(nix-store -qR "$SHELL_NIX_PATH")
88+
89+
# Verify the "devx" wrapper script is in the closure
90+
if ! echo "$CLOSURE_PATHS" | grep -q "devx$"; then
91+
echo "::error::No 'devx' wrapper found in closure"
92+
exit 1
93+
fi
94+
95+
DEVX_PATH=$(echo "$CLOSURE_PATHS" | grep "devx$" | head -1)
96+
echo "Found devx wrapper: $DEVX_PATH"
97+
98+
echo "::group::Smoke test"
99+
# Write smoke test commands to a temporary file — the devx
100+
# wrapper sources $1 as a script, it doesn't support -c.
101+
SMOKE_TEST=$(mktemp)
102+
cat > "$SMOKE_TEST" << 'TESTEOF'
103+
echo "GHC: $(ghc --version 2>/dev/null || echo N/A)"
104+
echo "Cabal: $(cabal --version 2>/dev/null | head -1 || echo N/A)"
105+
TESTEOF
106+
107+
# stdenv's setup.sh expects Nix builder runtime variables
108+
# (NIX_BUILD_TOP, out, etc.) that only exist inside nix build.
109+
# Provide them so the devx wrapper can source setup.sh.
110+
export NIX_BUILD_TOP="$(mktemp -d)"
111+
export TMPDIR="$NIX_BUILD_TOP"
112+
export TMP="$NIX_BUILD_TOP"
113+
export TEMP="$NIX_BUILD_TOP"
114+
export TEMPDIR="$NIX_BUILD_TOP"
115+
export NIX_STORE="/nix/store"
116+
export out="$NIX_BUILD_TOP/out"
117+
mkdir -p "$out"
118+
"$DEVX_PATH" "$SMOKE_TEST"
119+
echo "::endgroup::"
120+
121+
echo "Validation passed for ${{ matrix.job.short_name }}"

Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ if [ -n "\$VSCODE_IPC_HOOK_CLI" ]; then
2828
mkdir -p "\$(dirname "\$VSCODE_IPC_HOOK_CLI")"
2929
fi
3030
# This line is the one that actually brings DevX devshell ...
31-
source $(grep -m 1 -e '-env.sh$' store-paths.txt)
31+
source $(grep -m 1 -e 'devx$' store-paths.txt)
3232
EOF
3333

3434
# This enforce those settings in DevContainer whatever "Settings Sync" user preferences ...

extra/ghcr-upload.sh

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,14 @@ set -euox pipefail
77

88
# retry three times in case of intermittent network errors.
99
nix-store -r "$SHELL_NIX_PATH" || nix-store -r "$SHELL_NIX_PATH" || nix-store -r "$SHELL_NIX_PATH"
10-
#nix build ".#hydraJobs.${DEV_SHELL}" --show-trace --accept-flake-config
11-
nix-store --export $(nix-store -qR "$SHELL_NIX_PATH") | tee store-paths.txt | zstd -z8T8 >${DEV_SHELL}
12-
if [[ ! $(tail -n 1 store-paths.txt) =~ "devx" ]]; then exit 1; fi
10+
# Collect closure paths and export them as a zstd-compressed NAR archive.
11+
# Save the path list separately for verification — nix-store --export
12+
# produces binary output (contains null bytes) which can't be inspected
13+
# with text tools like tail/grep.
14+
CLOSURE_PATHS=$(nix-store -qR "$SHELL_NIX_PATH")
15+
nix-store --export $CLOSURE_PATHS | zstd -z8T8 >${DEV_SHELL}
16+
# Verify the environment script is included in the closure.
17+
# The wrapper script is always named "devx" for compatibility with
18+
# input-output-hk/actions/devx/action.yml which matches on that name.
19+
if ! echo "$CLOSURE_PATHS" | grep -q "devx$"; then exit 1; fi
1320
oras push ghcr.io/input-output-hk/devx:${DEV_SHELL} ${DEV_SHELL}

0 commit comments

Comments
 (0)