LXD-based development environment

Author: nooreldeenmansour

dev/DEV-GUIDE.md

@@ -0,0 +1,180 @@
+# authd Development Environment
+
+LXD VM for daily authd development. Provides full systemd, D-Bus, SSH, and GDM —
+enough to build, test, and exercise real PAM/NSS login flows without touching your
+host system. The host source tree is bind-mounted at `/workspace/authd`.
+
+## Prerequisites
+
+- LXD: `sudo snap install lxd && lxd init --auto && sudo usermod -aG lxd "$USER"` (logout/login)
+- SSH key: `ls ~/.ssh/id_ed25519.pub || ssh-keygen -t ed25519`
+- SPICE viewer (only needed for `lxc console --type=vga`): `sudo apt install virt-viewer`
+
+The SSH key is injected into the VM for key-based authentication, VS Code Remote SSH,
+and testing PAM login flows over SSH (`ssh user@domain.com@vm-ip`).
+
+## Quick Start
+
+```bash
+./dev/dev-env.sh up                    # Create VM + build authd (~20 min first time)
+./dev/dev-env.sh broker google \
+    --client-id ID --client-secret SEC \
+    --ssh-suffixes '@gmail.com'        # Configure broker
+ssh you@gmail.com@$(./dev/dev-env.sh ip)  # Test login from host
+```
+
+The `up` command provisions the VM, installs all toolchains (Go, Rust, protoc),
+and automatically builds + installs authd, PAM, and NSS modules. After `up`
+completes, the only manual step is configuring a broker with your IdP credentials.
+
+## Commands
+
+Global flags (before the subcommand, apply to all commands):
+`--name NAME` (default: authd-dev), `--release NAME` (default: noble), `--workspace PATH` (default: /workspace/authd).
+
+| Command | Description |
+|---------|-------------|
+| `up` | Create and provision VM (also restarts a stopped VM) |
+| `stop` | Stop VM (preserves state; restart with `up`) |
+| `down [--force]` | Stop and delete VM and profile |
+| `shell` | Direct shell via `lxc exec` (no PAM, always works) |
+| `ssh` | Connect via SSH (goes through PAM — for login testing) |
+| `status` | VM status and snapshots |
+| `snapshot <name>` / `restore <name>` | Manage snapshots |
+| `broker <variant> [opts]` | Configure credentials + install broker; `--rebuild` to recompile without touching credentials; `edit` subaction to open broker.conf |
+| `build [component]` | Fast rebuild + install a single component |
+| `validate` | Health-check authd stack (socket, PAM, NSS, brokers) |
+| `test [args]` | Run tests inside the VM (e.g. `--update-golden`, `--skip-external`) |
+| `logs [target]` | Tail logs (authd/google/msentraid/oidc/cloud-init) |
+| `exec <cmd>` | Run a command inside the VM (in workspace dir) |
+| `ip` | Print VM IP |
+
+## Iterative Development
+
+After the initial `install-authd`, use `build` from the **host**:
+
+```bash
+./dev/dev-env.sh build authd          # Daemon + proto regen + restart
+./dev/dev-env.sh build pam            # PAM modules (reconnect SSH to load)
+./dev/dev-env.sh build nss            # NSS module + ldconfig
+./dev/dev-env.sh build broker google  # Broker binary + restart
+./dev/dev-env.sh build all            # Full install-authd
+```
+
+To run tests inside the VM:
+
+```bash
+./dev/dev-env.sh test                                        # All tests with race detection (default)
+./dev/dev-env.sh test ./internal/brokers/...                 # Specific package
+./dev/dev-env.sh test --update-golden                        # Auto-update golden files
+./dev/dev-env.sh test --skip-external                        # Skip VHS (requires external tools)
+```
+
+Tail service logs from the host:
+
+```bash
+./dev/dev-env.sh logs authd            # authd daemon logs
+./dev/dev-env.sh logs google           # Google broker logs
+./dev/dev-env.sh logs cloud-init       # Cloud-init provisioning log
+```
+
+## Broker Setup
+
+```bash
+# Google (--issuer defaults to accounts.google.com)
+./dev/dev-env.sh broker google \
+    --client-id 843411...googleusercontent.com \
+    --client-secret GOCSPX-... \
+    --ssh-suffixes '@gmail.com'
+
+# Microsoft Entra ID
+./dev/dev-env.sh broker msentraid \
+    --issuer https://login.microsoftonline.com/TENANT_ID/v2.0 \
+    --client-id CLIENT_ID \
+    --ssh-suffixes '@yourdomain.com'
+
+# Generic OIDC (Keycloak, etc.)
+./dev/dev-env.sh broker oidc \
+    --issuer https://keycloak.example.com/realms/myrealm \
+    --client-id authd-client --client-secret SECRET \
+    --ssh-suffixes '*'
+```
+
+The `broker` command patches credentials, enables, and restarts the service automatically.
+Add `--rebuild` to recompile from source (e.g. after pulling broker changes).
+Use `broker <variant> edit` to open `broker.conf` directly in your editor.
+
+## Validation
+
+After installing authd (and optionally a broker), verify the full stack:
+
+```bash
+./dev/dev-env.sh validate
+```
+
+Checks: toolchain, authd.socket, PAM module, NSS config, broker registration.
+
+## Troubleshooting
+
+| Problem | Check |
+|---------|-------|
+| VM won't start | `lxc info authd-dev` |
+| Provisioning failed | `./dev/dev-env.sh logs cloud-init` or `lxc exec authd-dev -- cloud-init status` |
+| Build failed during `up` | `./dev/dev-env.sh exec ./dev/scripts/install-authd` to retry |
+| Wrong Go version | `which go` should be `/usr/local/go/bin/go`, not `/usr/bin/go` |
+| authd won't start | `sudo systemctl status authd.socket` (authd uses socket activation) |
+| SSH login rejects user | `ssh_allowed_suffixes_first_auth` must be set in broker.conf |
+| GDM console login password | `cat ~/.config/<vm-name>/vm-password` (default: `~/.config/authd-dev/vm-password`) |
+| VGA console freezes immediately | See [Known issue: VGA console on HiDPI](#known-issues) below |
+
+**Recovery from failed provisioning:** `./dev/dev-env.sh down --force && ./dev/dev-env.sh up`
+
+**Re-running install scripts:** `./dev/dev-env.sh exec ./dev/scripts/install-authd` is safe to re-run (idempotent config, rebuilds binaries).
+`broker` auto-detects whether a binary exists: if it does, only credentials are patched (no rebuild). Pass `--rebuild` to recompile from source.
+
+**Snapshots:** `up` creates two snapshots: `clean` (toolchain only, pre-build) and `installed` (authd + PAM + NSS built; no broker credentials — run `./dev/dev-env.sh broker <variant>` to configure).
+Use `./dev/dev-env.sh restore installed` to reset to a freshly built state.
+
+**Go/Rust versions** are auto-synced from `go.mod` and `authd-oidc-brokers/rust-toolchain.toml`
+at VM creation time. No manual version management needed.
+
+## GDM / VGA Console
+
+The VM runs GDM at `graphical.target` so you can test real GDM login flows.
+
+**Accessing the GDM screen:**
+```bash
+lxc console authd-dev --type=vga   # Opens SPICE viewer (requires virt-viewer on host)
+```
+
+The default login password is random-generated at VM creation time:
+```bash
+cat ~/.config/authd-dev/vm-password     # default VM name
+cat ~/.config/<vm-name>/vm-password     # custom --name
+```
+
+**Known issues:**
+
+| Issue | Status | Workaround |
+|-------|--------|------------|
+| **VGA console freezes on HiDPI displays (>1440p)** | [lxd#17606](https://github.com/canonical/lxd/issues/17606) — open bug, Ubuntu 25.10, LXD 6.6 | Lower host resolution to ≤1440p before opening console; or use LXD web UI (`lxc config show authd-dev` → connect on `:8443`) |
+| **VGA console drops on VM reboot** | [lxd#16184](https://github.com/canonical/lxd/issues/16184) — open, targeted for LXD 26.04 | Re-run `lxc console authd-dev --type=vga` after reboot |
+
+> **Wayland note:** `remote-viewer` (from `virt-viewer`) runs via XWayland on Wayland hosts.
+> The freeze issue is worse on Wayland + HiDPI due to SPICE resolution negotiation.
+> Using X11 (`DISPLAY=:0` session) or lowering resolution are the practical
+> workarounds until lxd#17606 is resolved.
+
+## File Layout
+
+```
+dev/
+├── dev-env.sh              # VM lifecycle + build (run from host)
+├── cloud-init.yaml         # VM provisioning template (Go, Rust, deps, GDM)
+├── DEV-GUIDE.md            # This file
+├── lib/
+│   └── common.sh           # Shared helpers (output, build, variant config)
+└── scripts/
+    ├── install-authd       # Build + install authd + PAM + NSS + configure (verbose mode)
+    └── install-broker      # Configure or build+install OIDC broker + D-Bus + systemd
+```

dev/cloud-init.yaml

@@ -0,0 +1,197 @@
+#cloud-config
+# authd Development Environment - Cloud-Init Configuration
+#
+# Single cloud-init user-data for the LXD VM dev environment.
+# Installs all build/test dependencies, Go, Rust, and a minimal GNOME desktop
+# for GDM login testing.
+#
+# Placeholders replaced at launch time by dev-env.sh:
+#   __SSH_PUBLIC_KEY__  - your SSH public key
+#   __GO_VERSION__      - Go toolchain version (from go.mod)
+#   __RUST_CHANNEL__    - Rust toolchain channel (from rust-toolchain.toml)
+#   __VM_PASSWORD__     - randomly generated VM password
+
+ssh_authorized_keys:
+  - __SSH_PUBLIC_KEY__
+
+package_update: true
+# Intentionally false (unlike e2e cloud-init which uses true) to speed up
+# VM provisioning. Dev VMs are short-lived and don't need full upgrades.
+package_upgrade: false
+
+packages:
+  # ── Build dependencies (from debian/control) ──
+  - protobuf-compiler
+  - pkgconf
+  - gcc
+  - make
+  - git
+  - libpam0g-dev
+  - libglib2.0-dev
+  - libpwquality-dev
+  - libc6-dev
+  - libssl-dev
+  - systemd-dev
+  # ── Test dependencies (from .github/workflows/qa.yaml) ──
+  - bubblewrap
+  - cracklib-runtime
+  - dbus
+  - openssh-server
+  - uidmap
+  - openssh-client
+  - apparmor-profiles
+  - gdm3
+  - gnome-shell
+  - spice-vdagent
+  # ── Dev ergonomics ──
+  - shellcheck            # Shell script linting (used in CI qa.yaml)
+  # ── Runtime / Utilities ──
+  - systemd-timesyncd
+  - curl
+  - wget
+  - jq
+  - vim
+  - tree
+  - ripgrep
+  - htop
+  - software-properties-common
+
+write_files:
+  # SSH config for authd PAM testing (matches e2e-tests/vm/cloud-init-template)
+  - path: /etc/ssh/sshd_config.d/authd.conf
+    owner: root:root
+    permissions: "0644"
+    content: |
+      UsePAM yes
+      Match User *@*
+          KbdInteractiveAuthentication yes
+
+  # PATH setup for Go and Rust toolchains
+  - path: /etc/profile.d/authd-dev.sh
+    owner: root:root
+    permissions: "0644"
+    content: |
+      # Go (installed to /usr/local/go from go.dev)
+      export PATH="/usr/local/go/bin:$HOME/go/bin:$PATH"
+      # Rust (rustup, installed per-user)
+      if [ -d "$HOME/.cargo/bin" ]; then
+        export PATH="$HOME/.cargo/bin:$PATH"
+      fi
+
+  # Suppress login banner noise (from e2e-tests/vm/cloud-init-template)
+  - path: /etc/skel/.hushlogin
+    content: ""
+
+  # dconf profile for the desktop session
+  - path: /etc/dconf/profile/user
+    content: |
+      user-db:user
+      system-db:local
+
+  # Prevent the desktop from suspending or blanking the screen
+  # (from e2e-tests/vm/cloud-init-template-questing.yaml)
+  - path: /etc/dconf/db/local.d/00-authd-dev
+    content: |
+      [org/gnome/settings-daemon/plugins/power]
+      sleep-inactive-ac-type='nothing'
+      sleep-inactive-ac-timeout=0
+
+      [org/gnome/desktop/session]
+      idle-delay=uint32 0
+
+runcmd:
+  # --- Configure swap (4GB) to prevent OOM during large Go builds ---
+  # msgraph-sdk-go/models requires >3GB during compilation; 4GB swap provides insurance.
+  - |
+    set -e
+    echo "Configuring 4GB swap file..."
+    fallocate -l 4G /swapfile
+    chmod 600 /swapfile
+    mkswap /swapfile
+    swapon /swapfile
+    echo '/swapfile none swap sw 0 0' >> /etc/fstab
+    echo 'vm.swappiness=10' >> /etc/sysctl.d/99-swap.conf
+    sysctl -p /etc/sysctl.d/99-swap.conf 2>/dev/null | grep swappiness || true
+    echo "Swap configured (4GB, swappiness=10 to prefer memory)"
+
+  # --- Install Go from go.dev ---
+  # System Go is too old for authd; version is injected from go.mod by dev-env.sh.
+  - |
+    set -e
+    GO_VERSION="__GO_VERSION__"
+    ARCH=$(dpkg --print-architecture)
+    case "$ARCH" in
+      amd64) GO_ARCH="amd64" ;;
+      arm64) GO_ARCH="arm64" ;;
+      *) echo "Unsupported architecture: $ARCH"; exit 1 ;;
+    esac
+    echo "Installing Go ${GO_VERSION} (${GO_ARCH})..."
+    wget -q "https://go.dev/dl/go${GO_VERSION}.linux-${GO_ARCH}.tar.gz" -O /tmp/go.tar.gz
+    # Verify download succeeded and is non-empty
+    if [ ! -s /tmp/go.tar.gz ]; then
+      echo "ERROR: Go download failed or is empty"; exit 1
+    fi
+    rm -rf /usr/local/go
+    tar -C /usr/local -xzf /tmp/go.tar.gz
+    rm -f /tmp/go.tar.gz
+    echo "Go installed: $(/usr/local/go/bin/go version)"
+
+  # --- Install Rust via rustup (for ubuntu user) ---
+  # RUSTUP_INIT_SKIP_PATH_CHECK: required because dh-cargo may pull in
+  # system Rust, which causes rustup to error.
+  # Toolchain channel is injected from rust-toolchain.toml by dev-env.sh.
+  - |
+    set -e
+    su - ubuntu -c 'export RUSTUP_INIT_SKIP_PATH_CHECK=yes && curl --proto "=https" --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain __RUST_CHANNEL__'
+    su - ubuntu -c '. "$HOME/.cargo/env" && rustup default __RUST_CHANNEL__ && echo "Rust: $(rustc --version), $(cargo --version)"'
+
+  # --- Set PATH in ubuntu's .bashrc for non-interactive SSH sessions ---
+  # /etc/profile.d only applies to login shells. Non-interactive SSH commands
+  # (ssh vm 'go build ...') need PATH set in .bashrc.
+  - |
+    cat >> /home/ubuntu/.bashrc <<'BASHRC'
+
+    # authd dev environment PATH (added by cloud-init)
+    export PATH="/usr/local/go/bin:$HOME/go/bin:$HOME/.cargo/bin:$PATH"
+    BASHRC
+
+  # --- Install Go protobuf tools (mirrors CONTRIBUTING.md method) ---
+  - |
+    su - ubuntu -c '
+      export PATH="/usr/local/go/bin:$HOME/go/bin:$HOME/.cargo/bin:$PATH"
+      cd /workspace/authd/tools
+      grep -o "_ \"[^\"]*\"" *.go | cut -d "\"" -f 2 | sort -u | xargs -r go install
+    '
+    echo "Go protobuf tools installed"
+
+  # --- AppArmor bubblewrap profile (required by tests) ---
+  - |
+    if [ -f /usr/share/apparmor/extra-profiles/bwrap-userns-restrict ]; then
+      ln -sf /usr/share/apparmor/extra-profiles/bwrap-userns-restrict /etc/apparmor.d/
+      apparmor_parser -r /etc/apparmor.d/bwrap-userns-restrict 2>/dev/null || true
+      echo "AppArmor bubblewrap profile loaded"
+    fi
+
+  # --- System hardening for dev comfort ---
+  - sed -i 's/^LOGIN_TIMEOUT.*/LOGIN_TIMEOUT 360/' /etc/login.defs
+  - systemctl mask systemd-networkd-wait-online.service || true
+  - systemctl mask apt-daily.service apt-daily.timer || true
+  - systemctl mask apt-daily-upgrade.service apt-daily-upgrade.timer || true
+  - systemctl mask unattended-upgrades.service || true
+  - rm -f /etc/apt/apt.conf.d/20auto-upgrades /etc/apt/apt.conf.d/50unattended-upgrades
+
+  # --- Desktop configuration ---
+  # graphical.target enables GDM for desktop login testing.
+  # SSH still works — GDM does not block SSH access.
+  - systemctl set-default graphical.target
+  - dconf update
+  - sed -i 's/enabled=1/enabled=0/' /etc/default/apport 2>/dev/null || true
+  - systemctl mask --now apport.service || true
+  # Password for GDM console login (SSH uses key auth).
+  # Generated randomly by dev-env.sh at VM creation time.
+  - echo 'ubuntu:__VM_PASSWORD__' | chpasswd
+
+  # --- Restart SSH to pick up authd PAM config ---
+  - systemctl restart ssh
+
+  - echo "=== authd development environment provisioning complete ==="