Skip to content

Commit ee2200d

Browse files
committed
refactor(installer): use a path: flake so the host stays gitignored
The generated per-host dir (hosts/<host>/) is gitignored because it holds secrets (install-answers.json, local.nix). A git flake only sees tracked files, so install.sh force-added the host with `git add --intent-to-add -f` just to make Nix see it — which staged those secrets and left the tree permanently dirty. Reference the repo as a `path:` flake instead, which copies the directory verbatim (gitignored files included), and drop the intent-add entirely. The host is now truly untracked: never staged, never in `git status`, secrets never enter the index. All install-time and on-box flake commands (nix eval/build, disko, nixos-install, coder-reset's nixos-rebuild) use path:/… refs. Also make the installed /etc/nixos-repo usable as a normal git checkout: chown root:wheel + group-writable so wheel users `git pull` / edit without sudo, and check out a `main` branch tracking origin (a baked image .git is a detached, branchless CI checkout, so `git pull` had no branch to pull). Docs (README, agents.md, hosts/incus-vm/README) updated for the path: rebuild command and the writable, branch-tracking repo; removed the stale intent-to-add / dirty-tree guidance.
1 parent 0072206 commit ee2200d

5 files changed

Lines changed: 73 additions & 61 deletions

File tree

README.md

Lines changed: 16 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,12 @@ NixOS configuration for Coder demo and workshop boxes.
2020
This repo is a Nix flake. `flake.nix` auto-discovers every subdirectory of
2121
`./hosts/` that contains a `default.nix` and exposes it as
2222
`nixosConfigurations.<folder-name>`. For normal install hosts the folder name
23-
is also the hostname, so `nixos-rebuild switch --flake .` auto-selects the
24-
right config on the running box. Adding a new host means creating a host
25-
folder, no flake.nix edit. The installer does this for you.
23+
is also the hostname, so `nixos-rebuild switch --flake path:/etc/nixos-repo`
24+
auto-selects the right config on the running box. Adding a new host means
25+
creating a host folder, no flake.nix edit. The installer does this for you.
26+
(The per-host dir is gitignored, so the box uses a `path:` flake ref — which
27+
copies the tree verbatim, gitignored files included — rather than a git flake,
28+
which would only see tracked files. See [Applying changes](#applying-changes).)
2629

2730
Hosts whose folder name starts with an underscore (`_appliance-iso`,
2831
`_appliance-disk`, `_installer-iso`) are image builds, not per-machine installs: they
@@ -61,7 +64,7 @@ sudo ./install.sh \
6164

6265
`./install.sh --help` lists everything. `--coder-admin-password-file PATH` and `--nixos-password-file PATH` read passwords from a file so they don't end up in shell history. `--no-reboot` skips the automatic reboot at the end.
6366

64-
The installer generates `hosts/<hostname>/{default.nix,local.nix,install-answers.json,facter.json}`, copies the repo into `/etc/nixos-repo` on the target, and symlinks `/etc/nixos/flake.nix`. After reboot, `nixos-rebuild switch` Just Works. Continue with [After install](#after-install).
67+
The installer generates `hosts/<hostname>/{default.nix,local.nix,install-answers.json,facter.json}`, copies the repo into `/etc/nixos-repo` on the target (owned `root:wheel` and checked out on `main`, so wheel users can `git pull` / edit it without sudo), and symlinks `/etc/nixos/flake.nix`. Rebuild with `sudo nixos-rebuild switch --flake path:/etc/nixos-repo`. Continue with [After install](#after-install).
6568

6669
> **Different partition layout?** Don't import `installer/bootstrap/disko-standard.nix`; drop your own disko config into the host folder instead. See [disko examples](https://github.com/nix-community/disko/tree/master/example).
6770
@@ -196,26 +199,24 @@ boot via `coder-init-admin.service`. After the reboot:
196199
defaults.
197200

198201
Subsequent edits to `coderd/` templates go out via `coder-template-sync`
199-
on every `sudo nixos-rebuild switch`.
202+
on every `sudo nixos-rebuild switch --flake path:/etc/nixos-repo`.
200203

201204
## Applying changes
202205

206+
The per-host dir (`hosts/<host>/`) is gitignored and stays untracked, so the
207+
box uses a `path:` flake ref (a plain git flake would not see it). The repo is
208+
owned `root:wheel`, so wheel users edit it and `git pull` without sudo.
209+
203210
```sh
204-
sudo nixos-rebuild switch # most changes
205-
sudo nixos-rebuild boot && sudo reboot # changes that touch the desktop stack
206-
207-
# Edited hosts/<host>/local.nix, install-answers.json, or facter.json? Re-mark intent-to-add:
208-
sudo git -C /etc/nixos-repo add --intent-to-add -f \
209-
hosts/<host>/local.nix \
210-
hosts/<host>/install-answers.json \
211-
hosts/<host>/facter.json
211+
sudo nixos-rebuild switch --flake path:/etc/nixos-repo # most changes
212+
sudo nixos-rebuild boot --flake path:/etc/nixos-repo && sudo reboot # desktop-stack changes
212213
```
213214

214215
## Updating nixpkgs / disko / facter
215216

216217
```sh
217218
sudo nix flake update --flake /etc/nixos-repo
218-
sudo nixos-rebuild switch
219+
sudo nixos-rebuild switch --flake path:/etc/nixos-repo
219220
```
220221

221222
This bumps `flake.lock` to the latest of each input.
@@ -274,7 +275,7 @@ Fully automated, no follow-up steps needed. The service:
274275
### Changing the admin password
275276

276277
1. Edit `hosts/<host>/install-answers.json`, update `initialUser.password` (or override `services.coder-nixos.initialUser.password` in `local.nix`).
277-
2. Run `sudo nixos-rebuild switch` to bake the new password into the service.
278+
2. Run `sudo nixos-rebuild switch --flake path:/etc/nixos-repo` to bake the new password into the service.
278279
3. Run `sudo systemctl start coder-reset` to wipe and re-bootstrap with the new password.
279280

280281
> If you need to change the password on a **live** deployment without a full wipe:

agents.md

Lines changed: 17 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Reference for AI coding agents and humans working on this repo.
88
|---|---|
99
| SSH | `ssh -i ~/.ssh/id_ed25519 coderbox@<TAILSCALE_IP>` |
1010
| Repo path | `/etc/nixos-repo/` (a Nix flake; `nixosConfigurations.<hostname>`) |
11-
| Git ops | `sudo git -C /etc/nixos-repo <command>` |
11+
| Git ops | `git -C /etc/nixos-repo <command>` (repo is `root:wheel`, wheel-writable) |
1212
| Coder URL | `http://coder-thinkcentre.local:3000` |
1313
| Coder token | stored in `/etc/coder/session-token` |
1414
| Coder binary | `coder` (in PATH via NixOS; resolves from nix store) |
@@ -27,16 +27,19 @@ sudo k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml ...
2727

2828
```sh
2929
# Service, package, or config changes — safe, non-destructive:
30-
cd /etc/nixos-repo && sudo nixos-rebuild switch --flake /etc/nixos-repo
30+
cd /etc/nixos-repo && sudo nixos-rebuild switch --flake path:/etc/nixos-repo
3131

3232
# Desktop stack (GNOME, GDM, Wayland) — must reboot:
33-
cd /etc/nixos-repo && sudo nixos-rebuild boot --flake /etc/nixos-repo && sudo reboot
33+
cd /etc/nixos-repo && sudo nixos-rebuild boot --flake path:/etc/nixos-repo && sudo reboot
3434
```
3535

3636
The repo is baked onto the box at **`/etc/nixos-repo`** (the canonical flake;
3737
`nixosConfigurations.<hostname>`, auto-selected by the running hostname). Edit
38-
files there, then rebuild. Always pass `--flake /etc/nixos-repo` (or `cd` into
39-
it and use `--flake .`) — see the `/etc/nixos` pitfall below.
38+
files there, then rebuild. Always pass `--flake path:/etc/nixos-repo`. The
39+
per-host dir (`hosts/<host>/`) is gitignored and untracked, so a `path:` flake
40+
ref is required — a plain git flake (`--flake /etc/nixos-repo` or `--flake .`)
41+
only sees tracked files and would not find the host. See the `/etc/nixos`
42+
pitfall below.
4043

4144
`nixos-rebuild switch` triggers the `coder-template-sync` activation script, which runs `terraform apply` in `coderd/` and pushes any template changes to Coder. The `/etc/coder/session-token` it needs is populated automatically by `coder-init-admin.service` on first boot, so this just works post-install.
4245

@@ -73,13 +76,14 @@ if something is off. Run `make fmt` to fix formatting locally.
7376

7477
## Git Workflow
7578

76-
All files in `/etc/nixos-repo/` are root-owned. Use `sudo git`:
79+
All files in `/etc/nixos-repo/` are owned `root:wheel` and group-writable, so
80+
wheel users run git without sudo:
7781

7882
```sh
7983
cd /etc/nixos-repo
80-
sudo git status
81-
sudo git add -p
82-
sudo git commit -m "feat: describe your change"
84+
git status
85+
git add -p
86+
git commit -m "feat: describe your change"
8387
# Don't push unless explicitly asked
8488
```
8589

@@ -122,7 +126,7 @@ sudo terraform apply \
122126
-var="coder_url=http://localhost:3000" \
123127
-var="coder_session_token=$(sudo cat /etc/coder/session-token)" \
124128
-var="hostname=coder-thinkcentre" \
125-
-var="version_name=$(sudo git -C /etc/nixos-repo rev-parse --short HEAD)"
129+
-var="version_name=$(git -C /etc/nixos-repo rev-parse --short HEAD)"
126130
```
127131

128132
## nook-android Template
@@ -197,8 +201,8 @@ sudo k3s kubectl describe pod -n coder-workspaces <pod-name>
197201
- **Tailscale auth doesn't re-run**`tailscale-autoauth` has `RemainAfterExit = true`. If you change auth key config, run `sudo systemctl restart tailscale-autoauth`.
198202
- **Template sync skips**, if `/etc/coder/session-token` is empty, the activation script exits cleanly. The token is auto-populated by `coder-init-admin.service`; if it's missing, check `journalctl -u coder-init-admin`.
199203
- **`coder` binary path** — the binary is in PATH via NixOS environment; don't hardcode nix store paths in scripts (they change with every package update).
200-
- **`--flake /etc/nixos` fails**`/etc/nixos` is a plain dir holding only a `flake.nix` *symlink* into `/etc/nixos-repo`. Nix follows the symlink into the store but can't find the sibling files (configuration.nix, hosts/, nixos/), dying with `path '/nix/store/...-source/etc/nixos-repo/flake.nix' does not exist`. Always rebuild against the real tree: `--flake /etc/nixos-repo` (or `cd /etc/nixos-repo && nixos-rebuild switch --flake .`).
201-
- **`Git tree '/etc/nixos-repo' is dirty` warning** — harmless. `hosts/<host>/{local.nix,install-answers.json,facter.json}` are gitignored and intent-to-added by the installer, so the tree always reads "dirty". After editing them, re-mark intent-to-add so the flake sees them: `sudo git -C /etc/nixos-repo add --intent-to-add -f hosts/<host>/local.nix hosts/<host>/install-answers.json hosts/<host>/facter.json`.
204+
- **`--flake /etc/nixos` fails**`/etc/nixos` is a plain dir holding only a `flake.nix` *symlink* into `/etc/nixos-repo`. Nix follows the symlink into the store but can't find the sibling files (configuration.nix, hosts/, nixos/), dying with `path '/nix/store/...-source/etc/nixos-repo/flake.nix' does not exist`. Always rebuild against the real tree with a `path:` ref: `--flake path:/etc/nixos-repo`.
205+
- **Host not found / `does not provide attribute 'nixosConfigurations.<host>'`** — the per-host dir is gitignored and untracked, so a git flake (`--flake /etc/nixos-repo` or `--flake .`) can't see it. Use `--flake path:/etc/nixos-repo`, which copies the tree verbatim (gitignored files included). This is also why nothing intent-adds the host and `git status` stays clean.
202206
- **ScreenConnect blank screen** — the box now runs GNOME on Wayland (GDM), and GNOME 49 dropped the Xorg session, so there is no X11 desktop to fall back to. ScreenConnect reaches `DISPLAY=:0` through XWayland (see `nixos/screenconnect.nix`) but **cannot screen-capture the Wayland compositor** through it, so the remote view may be black/blank. Capturing the GNOME session needs a Wayland-aware path (PipeWire/portal, e.g. `gnome-remote-desktop`); the X11 agent will connect but not mirror the desktop.
203207

204208
## Wildcard App Access (TODO)
@@ -212,7 +216,7 @@ sudo k3s kubectl describe pod -n coder-workspaces <pod-name>
212216
## File Layout (agent-relevant paths)
213217

214218
```
215-
/etc/nixos-repo/ # repo root (a Nix flake; sudo git required)
219+
/etc/nixos-repo/ # repo root (a Nix flake; root:wheel, wheel-writable)
216220
flake.nix # entry point: nixosConfigurations.<host> per machine
217221
flake.lock # pinned nixpkgs / disko
218222
configuration.nix # shared NixOS config (edit here for services/packages)

configuration.nix

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,14 @@
22
# Per-host modules (hardware-configuration.nix or facter.json, optional disko,
33
# local.nix) live under ./hosts/<host>/; everything else lives here.
44
#
5-
# Apply: sudo nixos-rebuild switch (packages/services only)
6-
# sudo nixos-rebuild boot + sudo reboot (anything touching desktop/display stack)
5+
# Apply: sudo nixos-rebuild switch --flake path:/etc/nixos-repo (packages/services only)
6+
# sudo nixos-rebuild boot --flake path:/etc/nixos-repo + sudo reboot (desktop/display stack)
77
#
88
# First-time setup and the live-USB install walkthrough are in ./README.md.
99
# This file expects a flake (./flake.nix) to assemble the configuration via
10-
# nixosConfigurations.<hostname>, so `nixos-rebuild switch` should resolve
11-
# through /etc/nixos/flake.nix (symlinked to /etc/nixos-repo/flake.nix).
10+
# nixosConfigurations.<hostname>. Rebuilds use a `path:` flake ref
11+
# (path:/etc/nixos-repo) rather than the plain git flake so the gitignored
12+
# per-host dir (hosts/<host>/, holding secrets) is visible to Nix.
1213
#
1314
# Per-host local.nix lives at hosts/<host>/local.nix and is gitignored.
1415

@@ -797,7 +798,7 @@ in
797798
# 10. Re-run nixos-rebuild switch to push templates via coder-template-sync
798799
echo "--- running nixos-rebuild switch (template sync)"
799800
/run/current-system/sw/bin/nixos-rebuild switch \
800-
--flake /etc/nixos-repo 2>&1 \
801+
--flake path:/etc/nixos-repo 2>&1 \
801802
| ${pkgs.gnused}/bin/sed 's/^/[coder-reset] /'
802803
803804
echo ""

hosts/incus-vm/README.md

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -92,11 +92,12 @@ cp /etc/nixos-repo/hosts/incus-vm/incus-vm.nix \
9292
/etc/nixos-repo/hosts/$HOSTNAME/incus-vm.nix
9393

9494
# For bare-metal — write your own default.nix or copy from another host.
95-
96-
# Stage the files — the flake's builtins.readDir only sees tracked files.
97-
git -C /etc/nixos-repo add hosts/$HOSTNAME/
9895
```
9996

97+
The host dir is gitignored (it holds per-host secrets) and stays untracked.
98+
Rebuilds below use a `path:` flake ref, which sees gitignored files, so there's
99+
nothing to `git add`.
100+
100101
> **`/etc/nixos/coder.nix`:** The copied `default.nix` does **not** import this
101102
> file. It only exists on VMs that are *also* running as a coder-agent workspace
102103
> (i.e. the `incus-nixos` template writes it). On a pure box host it won't be
@@ -138,9 +139,6 @@ gitignored and must be created manually:
138139
```sh
139140
cp /etc/nixos-repo/installer/bootstrap/local.nix.example \
140141
/etc/nixos-repo/hosts/$HOSTNAME/local.nix
141-
142-
# Mark it so the flake's builtins.readDir can see it without committing it.
143-
git -C /etc/nixos-repo add --intent-to-add -f hosts/$HOSTNAME/local.nix
144142
```
145143

146144
Edit `hosts/$HOSTNAME/local.nix` and at minimum set:
@@ -178,7 +176,7 @@ EOF
178176
### 6. Apply
179177

180178
```sh
181-
nixos-rebuild switch --flake /etc/nixos-repo#$(hostname -s) --impure
179+
nixos-rebuild switch --flake path:/etc/nixos-repo#$(hostname -s) --impure
182180
```
183181

184182
`--impure` is required because `/etc/nixos/incus.nix` lives outside the flake
@@ -214,5 +212,5 @@ Open that URL in a browser, create the admin user, then log in with the CLI:
214212
CODER_URL=http://localhost:3000 coder login http://localhost:3000
215213
```
216214

217-
Once logged in, run `sudo nixos-rebuild switch` again to push templates via
218-
`template-sync`.
215+
Once logged in, run `sudo nixos-rebuild switch --flake path:/etc/nixos-repo#$(hostname -s) --impure`
216+
again to push templates via `template-sync`.

install.sh

Lines changed: 27 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -695,23 +695,17 @@ if [[ ! -f "$HOST_DIR/facter.json" ]]; then
695695
echo " wrote hosts/$HOSTNAME_ARG/facter.json"
696696
fi
697697

698-
# A git path flake ignores untracked files, so the freshly written host files
699-
# must be intent-to-added for the flake to see them (local.nix and
700-
# install-answers.json are gitignored, so force-add them). Only meaningful when
701-
# REPO_DIR is a git repo; the ISO writable copy may have no .git (a non-git path
702-
# flake already sees every file), so skip.
703-
if git -C "$REPO_DIR" rev-parse --is-inside-work-tree >/dev/null 2>&1; then
704-
git -C "$REPO_DIR" add --intent-to-add -f \
705-
"hosts/$HOSTNAME_ARG/default.nix" \
706-
"hosts/$HOSTNAME_ARG/facter.json" \
707-
"hosts/$HOSTNAME_ARG/install-answers.json" \
708-
"hosts/$HOSTNAME_ARG/local.nix" >/dev/null
709-
fi
698+
# The host dir (hosts/$HOSTNAME_ARG/) is gitignored and stays untracked — it
699+
# holds per-box secrets (install-answers.json, local.nix). Every flake command
700+
# below references the repo as a `path:` flake, which copies the directory
701+
# verbatim (gitignored files included) instead of using the git tree, so the
702+
# host is visible without intent-to-add / force-staging and its secrets never
703+
# enter the git index.
710704

711705
# ── Validate ───────────────────────────────────────────────────────────────
712706
echo " validating flake ..."
713707
nix --extra-experimental-features 'nix-command flakes' \
714-
eval "$REPO_DIR#nixosConfigurations.${HOSTNAME_ARG}.config.system.build.toplevel.drvPath" \
708+
eval "path:$REPO_DIR#nixosConfigurations.${HOSTNAME_ARG}.config.system.build.toplevel.drvPath" \
715709
>/dev/null
716710

717711
# ── Partition + format + mount ─────────────────────────────────────────────
@@ -720,7 +714,7 @@ echo "=== Partitioning $DISK_ARG via disko ==="
720714
# Use the flake's pinned disko (one nixpkgs source for the whole install).
721715
nix --extra-experimental-features 'nix-command flakes' \
722716
run "$REPO_DIR#disko" -- \
723-
--mode disko --flake "$REPO_DIR#${HOSTNAME_ARG}"
717+
--mode disko --flake "path:$REPO_DIR#${HOSTNAME_ARG}"
724718

725719
mountpoint -q /mnt || {
726720
echo "disko did not mount /mnt" >&2
@@ -745,8 +739,22 @@ echo "=== Copying repo into /mnt/etc/nixos-repo ==="
745739
mkdir -p /mnt/etc/nixos-repo
746740
cp -a "$REPO_DIR/." /mnt/etc/nixos-repo/
747741

748-
# Symlink /etc/nixos/flake.nix so plain `nixos-rebuild switch` finds the
749-
# config after reboot.
742+
# Make the repo writable by the login user (wheel group) so they can `git pull`
743+
# and edit config in place without sudo. Also ensure a `main` branch is checked
744+
# out and tracking origin, so `git pull` works out of the box: a baked image's
745+
# .git is a detached, branchless CI checkout, and a fresh live-USB clone is
746+
# already on main (this is a harmless no-op there).
747+
chown -R root:wheel /mnt/etc/nixos-repo
748+
chmod -R g+w /mnt/etc/nixos-repo
749+
if [[ -d /mnt/etc/nixos-repo/.git ]]; then
750+
git -C /mnt/etc/nixos-repo checkout -B main >/dev/null 2>&1 || true
751+
git -C /mnt/etc/nixos-repo config branch.main.remote origin
752+
git -C /mnt/etc/nixos-repo config branch.main.merge refs/heads/main
753+
fi
754+
755+
# Symlink /etc/nixos/flake.nix so tooling that looks in /etc/nixos finds the
756+
# flake. Rebuilds must use `--flake path:/etc/nixos-repo` so the gitignored
757+
# per-host dir is visible (a plain git flake would not see it).
750758
mkdir -p /mnt/etc/nixos
751759
ln -sf /etc/nixos-repo/flake.nix /mnt/etc/nixos/flake.nix
752760

@@ -782,7 +790,7 @@ if [[ ${CODER_BOX_FROM_IMAGE:-0} == "1" ]]; then
782790
SYSTEM_TOPLEVEL=$(nix --extra-experimental-features 'nix-command flakes' \
783791
build --no-link --print-out-paths \
784792
--option download-buffer-size 268435456 \
785-
"/mnt/etc/nixos-repo#nixosConfigurations.${HOSTNAME_ARG}.config.system.build.toplevel")
793+
"path:/mnt/etc/nixos-repo#nixosConfigurations.${HOSTNAME_ARG}.config.system.build.toplevel")
786794
[[ -n $SYSTEM_TOPLEVEL ]] || {
787795
echo "failed to build system closure" >&2
788796
exit 1
@@ -802,7 +810,7 @@ else
802810
echo "=== Running nixos-install ==="
803811
echo " (closure builds into /mnt/nix/store; no tmpfs OOM risk)"
804812
nixos-install \
805-
--flake "/mnt/etc/nixos-repo#${HOSTNAME_ARG}" \
813+
--flake "path:/mnt/etc/nixos-repo#${HOSTNAME_ARG}" \
806814
--no-channel-copy \
807815
--no-root-passwd \
808816
--option download-buffer-size 268435456
@@ -820,7 +828,7 @@ echo " http://${HOSTNAME_ARG}.local:3000 (direct LAN access)"
820828
echo " the *.try.coder.app URL itself is written to /etc/motd on first boot once coder.service is up"
821829
echo
822830
echo "Optional after first login:"
823-
echo " - Update the box: cd /etc/nixos-repo && sudo git pull && sudo nixos-rebuild switch"
831+
echo " - Update the box: cd /etc/nixos-repo && git pull && sudo nixos-rebuild switch --flake path:/etc/nixos-repo"
824832
echo
825833

826834
if [[ $NO_REBOOT -eq 0 ]]; then

0 commit comments

Comments
 (0)