lucernae
diff --git a/‎.devcontainer/nixos-local/README.md‎
Lines changed: 83 additions & 0 deletions b/‎.devcontainer/nixos-local/README.md‎
Lines changed: 83 additions & 0 deletions
diff --git a/‎.devcontainer/nixos-local/TROUBLESHOOTING.md‎
Lines changed: 247 additions & 0 deletions b/‎.devcontainer/nixos-local/TROUBLESHOOTING.md‎
Lines changed: 247 additions & 0 deletions
diff --git a/‎.devcontainer/nixos-local/devcontainer.json‎
Lines changed: 29 additions & 0 deletions b/‎.devcontainer/nixos-local/devcontainer.json‎
Lines changed: 29 additions & 0 deletions
diff --git a/‎.devcontainer/nixos-local/docker-compose.yml‎
Lines changed: 33 additions & 0 deletions b/‎.devcontainer/nixos-local/docker-compose.yml‎
Lines changed: 33 additions & 0 deletions
@@ -0,0 +1,83 @@
+# NixOS Devcontainer - Local Test
+
+This directory contains a devcontainer configuration to test the pre-built NixOS image from GHCR.
+
+## Usage
+
+### Option 1: VS Code
+
+1. Open this repository in VS Code
+2. Press `Ctrl+Shift+P` (or `Cmd+Shift+P` on Mac)
+3. Type "Dev Containers: Open Folder in Container"
+4. Select `.devcontainer/nixos-local`
+
+### Option 2: Command Line with Devcontainer CLI
+
+```bash
+devcontainer up --workspace-folder . --config .devcontainer/nixos-local/devcontainer.json
+```
+
+### Option 3: Docker Compose (Manual)
+
+```bash
+cd .devcontainer/nixos-local
+docker-compose up -d
+docker-compose exec devcontainer bash
+```
+
+## Image Details
+
+- **Source**: `ghcr.io/lucernae/devcontainer-nix:nixos--nixos-25.11`
+- **Architecture**: Multi-arch (x86_64-linux, aarch64-linux)
+- **Channel**: nixos-25.11
+- **Features**:
+  - Full NixOS system with systemd
+  - Nix flakes support
+  - Docker client
+  - Pre-configured with vscode user
+
+## Testing Checklist
+
+Once inside the container, verify:
+
+```bash
+# Check NixOS version
+nixos-version
+
+# Check Nix version
+nix --version
+
+# Check systemd is running
+systemctl status
+
+# Test Docker
+docker --version
+
+# Check vscode user exists
+id vscode
+
+# Test Nix flakes
+nix flake --help
+```
+
+## Troubleshooting
+
+### Container fails to start
+
+Make sure you're running in privileged mode and have cgroup v2 support:
+
+```bash
+docker info | grep -i cgroup
+```
+
+### Permission denied errors
+
+The container runs as root by default for NixOS initialization. The `vscode` user is available for development work.
+
+### Image not found
+
+If the image is not found, it may not have been pushed yet. Check the GitHub Actions workflow status or run:
+
+```bash
+docker pull ghcr.io/lucernae/devcontainer-nix:nixos--nixos-25.11
+```
@@ -0,0 +1,247 @@
+# NixOS Devcontainer Troubleshooting Guide
+
+## Rule 1: Final Test Procedure
+
+**All final tests MUST use devcontainer CLI and observe until fully loaded.**
+
+```
+Goal Condition: FHS is created and observable
+- /etc/passwd exists
+- /etc/group exists  
+- /bin/sh exists
+- systemd is running
+```
+
+### Test Procedure
+
+```bash
+# 1. Clean up any existing containers
+docker rm -f nixos-local-devcontainer-1 2>/dev/null
+
+# 2. Run devcontainer CLI
+nix-shell -p devcontainer --run "devcontainer up --workspace-folder . --config .devcontainer/nixos-local/devcontainer.json"
+
+# 3. OBSERVE until container is fully loaded:
+#    - Wait for "Dev Containers: Finished" or similar success message
+#    - Check systemd: docker exec nixos-local-devcontainer-1 systemctl is-system-running
+#    - Check FHS: docker exec nixos-local-devcontainer-1 ls -la /etc/passwd /bin/sh
+
+# 4. Goal condition check:
+docker exec nixos-local-devcontainer-1 test -f /etc/passwd && echo "✓ /etc/passwd exists"
+docker exec nixos-local-devcontainer-1 test -f /etc/group && echo "✓ /etc/group exists"
+docker exec nixos-local-devcontainer-1 test -f /bin/sh && echo "✓ /bin/sh exists"
+docker exec nixos-local-devcontainer-1 systemctl is-system-running | grep -q running && echo "✓ systemd running"
+```
+
+## Rule 2: Proof of Concept First
+
+**When troubleshooting, do NOT wait for GitHub Actions rebuild.**
+
+1. Use existing GHCR image as base
+2. Add modifications in local Dockerfile
+3. Test locally with devcontainer CLI first
+4. Only after POC works, add to container-layeredImage.nix for permanent fix
+
+### Workflow
+
+```
+GitHub Actions (permanent fix)  -------->  GHCR Image
+       ↑                                    ↑
+       |                                    |
+       | (after POC works)                 | (for testing)
+       |                                    |
+       +--------  container-layeredImage  <--+
+                          ↑
+                          |
+                   (add fix here)
+                          |
+                  docker-compose.yml
+                  + Dockerfile (POC)
+                          ↓
+                   devcontainer CLI
+```
+
+## Rule 3: Never Copy Binaries - Use Symlinks from Nix Store
+
+**The base image already has all binaries in the nix store. Never copy binaries - create symlinks instead.**
+
+### Why:
+- Copying binaries bloats the image
+- Nix store has all needed binaries (bash, coreutils, etc.)
+- Just create symlinks to the nix store paths
+
+### Finding Nix Store Paths
+
+```bash
+# Find bash in nix store
+docker run --rm ghcr.io/lucernae/devcontainer-nix:nixos--nixos-25.11---x86_64-linux \
+  find /nix/store -name bash -type f 2>/dev/null | head -5
+
+# Or export and check
+docker create --name temp <image>
+docker export temp | tar -tf - | grep "bin/bash"
+docker rm temp
+```
+
+## Rule 4: Original NixOS Systemd Must Be PID 1
+
+**When overriding Docker's entrypoint, the original NixOS systemd entrypoint MUST be PID 1.**
+
+### Why:
+- NixOS expects systemd as PID 1
+- Systemd handles initialization, services, cgroups
+- Blocking the init process breaks the system
+
+### Correct Pattern (Fork and Wait):
+
+```yaml
+# WRONG - blocks systemd
+entrypoint: ["bash", "-c", "wait_for_ready && /usr/sbin/init"]
+
+# CORRECT - start systemd in background, wait for it
+entrypoint: [
+  "bash", "-c",
+  "/usr/sbin/init &  # start systemd as PID 1
+   PID1=$!
+   wait_for_vscode_ready
+   wait $PID1"
+]
+```
+
+### Alternative: Use postCreateCommand
+
+Instead of blocking entrypoint, use devcontainer's `postCreateCommand`:
+
+```json
+{
+  "postCreateCommand": "bash -c 'systemctl is-system-running --wait && echo Ready'"
+}
+```
+
+This runs AFTER container is created and systemd is already running.
+
+## Problem: VS Code Dev Containers fails to connect
+
+### Error: `unable to find user root: no matching entries in passwd file`
+
+**Root Cause:**
+- VS Code runs `uname -m` as a probe to check system architecture
+- This requires a valid `/etc/passwd` entry for root
+- The NixOS activation hasn't run yet when VS Code tries to connect (systemd hasn't fully started)
+
+### Why this happens:
+1. Docker starts container with `/usr/sbin/init` (NixOS init)
+2. NixOS init runs activation scripts (creates `/etc/passwd`, `/etc/group`, sets up `/bin/sh`)
+3. After activation, systemd starts as PID 1
+4. **BUT** VS Code tries to connect BEFORE systemd is ready
+5. VS Code needs `/etc/passwd` to run its probe commands
+
+## Solutions
+
+### Solution 1: Add placeholder /etc/passwd in image (Permanent Fix)
+
+The VS Code probe runs BEFORE any devcontainer commands (even `onCreateCommand`). There is no built-in way to delay the probe.
+
+**Therefore, the solution is to add placeholder /etc/passwd and /etc/group to the image at build time.**
+
+Add passwd/group files in the NixOS image build (in container-layeredImage.nix):
+
+```nix
+fhsEtc = pkgs.runCommand "fhs-etc" { } ''
+  mkdir -p $out/etc
+  echo "root:x:0:0:root:/root:/run/current-system/sw/bin/zsh" > $out/etc/passwd
+  echo "root:x:0:" > $out/etc/group
+  echo "vscode:x:1000:" >> $out/etc/group
+'';
+```
+
+Include in `contents` array of streamLayeredImage.
+
+### Solution 2: Start container first, then use devcontainer CLI (Workaround)
+
+This is a practical workaround - start the container first, then use devcontainer CLI:
+
+```bash
+# 1. Start container with docker-compose
+docker compose up -d
+
+# 2. Wait for systemd to boot
+sleep 10
+
+# 3. Now run devcontainer CLI - it will attach to running container
+devcontainer up --workspace-folder . --config .devcontainer/nixos-local/devcontainer.json
+```
+
+This works because:
+1. docker-compose starts the container and waits for systemd to boot
+2. When systemd is ready, /etc/passwd and /etc/group exist
+3. devcontainer CLI then connects to the already-running container
+4. VS Code probe succeeds because FHS files exist
+
+### Solution 3: Use entrypoint wrapper (Alternative)
+
+Use a custom entrypoint that waits for systemd (Rule 4):
+
+```yaml
+services:
+  devcontainer:
+    entrypoint: [
+      "bash", "-c",
+      "/usr/sbin/init &  # start systemd as PID 1
+       PID1=$!
+       # wait for systemd to be ready
+       for i in 1 2 3 4 5 6 7 8 9 10; do
+         sleep 2
+         if systemctl is-system-running 2>/dev/null | grep -q running; then
+           break
+         fi
+       done
+       wait $PID1"
+    ]
+```
+
+### Solution 2: Add placeholder /etc/passwd to image (Permanent Fix)
+
+Add passwd/group files in the NixOS image build (in container-layeredImage.nix):
+
+```nix
+fhsEtc = pkgs.runCommand "fhs-etc" { } ''
+  mkdir -p $out/etc
+  echo "root:x:0:0:root:/root:/run/current-system/sw/bin/zsh" > $out/etc/passwd
+  echo "root:x:0:" > $out/etc/group
+  echo "vscode:x:1000:" >> $out/etc/group
+'';
+```
+
+Include in `contents` array of streamLayeredImage.
+
+**IMPORTANT: DO NOT COPY /etc/ in Dockerfile!**
+- Copying /etc/ files to the image will break NixOS activation
+- The /etc/ must be generated by NixOS activation at runtime
+- Placeholder files must be added at Nix expression level (container-layeredImage.nix)
+
+## Current Status
+
+- ✅ systemd boots correctly with `--cgroupns=host`
+- ✅ Nix and NixOS work inside the container  
+- ✅ docker-compose works directly with original GHCR image
+- ❌ VS Code Dev Containers fails (needs passwd before systemd ready)
+
+## Debugging
+
+```bash
+# Check systemd status (must show "running")
+docker exec nixos-local-devcontainer-1 systemctl is-system-running
+
+# Check activation logs
+docker logs nixos-local-devcontainer-1
+
+# Check FHS files exist
+docker exec nixos-local-devcontainer-1 ls -la /etc/passwd
+docker exec nixos-local-devcontainer-1 ls -la /etc/group  
+docker exec nixos-local-devcontainer-1 ls -la /bin/sh
+
+# Check Nix works
+docker exec nixos-local-devcontainer-1 nix --version
+docker exec nixos-local-devcontainer-1 nixos-version
+```
@@ -0,0 +1,29 @@
+// NixOS Devcontainer - Local Test Configuration
+// Uses pre-built image from GHCR
+{
+	"name": "NixOS Devcontainer (Local Test)",
+
+	"dockerComposeFile": [
+		"docker-compose.yml"
+	],
+
+	"overrideCommand": false,
+	"service": "devcontainer",
+	"workspaceFolder": "/workspace",
+
+	"postCreateCommand": "bash -c \"systemctl is-system-running --wait || true && nix-channel --update && echo 'NixOS test container ready!'\"",
+
+	"remoteUser": "root",
+
+	"customizations": {
+		"vscode": {
+			"extensions": [
+				"ms-azuretools.vscode-docker",
+				"arrterian.nix-env-selector",
+				"bbenoist.Nix",
+				"jnoortheen.nix-ide",
+				"brettm12345.nixfmt-vscode"
+			]
+		}
+	}
+}
@@ -0,0 +1,33 @@
+version: '3'
+services:
+  devcontainer:
+    # Using the pre-built NixOS image from GHCR
+    image: ghcr.io/lucernae/devcontainer-nix:nixos--nixos-25.11
+    privileged: true
+    tty: true
+    cgroup: host
+    
+    volumes:
+      - ../../:/workspace:cached
+      - /var/run/docker.sock:/var/run/docker.sock
+      - /sys/fs/cgroup:/sys/fs/cgroup:rw
+    
+    tmpfs:
+      - /run:exec,mode=777
+      - /run/lock:exec,mode=777
+
+    hostname: nixos-devcontainer
+
+    environment:
+      container: "docker"
+      LD_LIBRARY_PATH: /lib:/run/current-system/sw/lib
+      NIX_REMOTE: "daemon"
+      PATH: "/bin:/usr/bin:/usr/sbin:/run/current-system/sw/bin/"
+      TMPDIR: "/cont/tmp"
+      VSCODE_SERVER_CUSTOM_GLIBC_LINKER: "true"
+      
+    cap_add:
+      - SYS_ADMIN
+      - CAP_NET_ADMIN
+
+    stop_signal: SIGRTMIN+3