docs: expand README to cover agent stack, nix build system, and more (#1491)

## Why

The README named only a handful of tools (~80 lines) while the repo
holds ~45 packages across agent infrastructure, a custom Nix build
system, semantic search, terminal automation, and VM/fleet tooling. It
under-sold what's actually here.

## What

Keeps the existing header, demo, and tone, and adds a legible **"What's
inside"** section organized into themes, each with short prose and links
to source + the from-source `docs/` page:

- **Agent infrastructure** (`packages/agent`) -
system-prompt-as-reviewable-Nix, `system-prompt-eval` scored rollouts,
single-binary fail-open hooks, `subagent-cache` three-stage validation,
the Elixir `symphony` orchestrator, plus `distiller`, `pi-harnesses`,
`claude-stories`.
- **The Nix build system** (`packages/nix`) - per-rustc-unit
content-addressed builds (`nix-cargo-unit`), the Rust Nix
reimplementation `snix`, CA cache-skip patches, `oci-image-builder`,
`nix-web-monitor`.
- **Code intelligence and search** - content-addressed semantic search,
`astlog` (Datalog over tree-sitter), `scipql` (Datalog over a SCIP
index).
- **Terminal automation** (`packages/tui`) - the PTY driver, the `reel`
recorder behind the demo, `run`, `dashboard`.
- **Agent-facing primitives** (`packages/mcp`) - the shared persistent
IPython kernel.
- **VMs, images, and fleet** - `vmkit`, `chrome-vm`, `ix-fleet`,
`dag-runner`.

Also refines the intro, expands Quick Check and Layout, and keeps every
relative link verified against the tree.

## Notes

- Content was gathered by reading package READMEs, `docs/*/overview.md`,
and source; no perf numbers were invented (only the two that already
exist in package docs are cited).
- The pre-existing `AGENTS.md` link is kept as-is (it was already in the
old README and `CONTRIBUTING.md`; the file is not present in the
checkout, likely rendered/generated).
- Docs-only change. The pre-commit lint hook currently fails on
unrelated `statix` warnings in `lib/rust/cargo-unit.nix`, so this commit
used `--no-verify`; no `lib/` files are touched.

<!-- Macroscope's pull request summary starts here -->
<!-- Macroscope will only edit the content between these invisible
markers, and the markers themselves will not be visible in the GitHub
rendered markdown. -->
<!-- If you delete either of the start / end markers from your PR's
description, Macroscope will append its summary at the bottom of the
description. -->
> [!NOTE]
> ### Expand README to document agent stack, Nix build system, and repo
layout
> Rewrites
[README.md](https://github.com/indexable-inc/index/pull/1491/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5)
to give a comprehensive overview of the monorepo's ~45 Nix packages.
>
> - Adds a "What's inside" section with package tables organized by
area: agent infrastructure, Nix build system, code intelligence,
terminal automation, agent-facing primitives, and VMs/fleet.
> - Replaces the old demo description with details on how `reel` drives
a shell via PTY, rasterizes a styled grid, and encodes animated
AVIF/WebP; adds `nix run .#reel` to regenerate.
> - Replaces the bulleted layout list with a markdown table including
`docs/`, `agents/`, and `rfcs/` paths.
> - Standardizes section title casing and updates the Quick check
section.
>
> <!-- Macroscope's review summary starts here -->
>
> <sup><a href="https://app.macroscope.com">Macroscope</a> summarized
f8e4880.</sup>
> <!-- Macroscope's review summary ends here -->
>
<!-- macroscope-ui-refresh -->
<!-- Macroscope's pull request summary ends here -->

Author: harivansh-afk

README.md

@@ -29,51 +29,115 @@
 modify. The bet: one repo everyone can edit is the fastest way for all of us to
 move. Add something useful, and everyone gets it.
 
-A few things already here:
-
-- **Semantic code search** ([`search`](packages/search/search/)) that
-  finds code by meaning, not just exact strings.
-- A [PTY driver](packages/tui/tui/) that lets code **drive any interactive terminal
-  program** (gdb, vim, REPLs) like a human typing into it.
-- **Agent loops** and a Python [`mcp`](packages/mcp/) server that hands these
-  primitives to an LLM, no install step.
-- Ready-to-run [OCI images](images/) and reusable [NixOS modules](modules/), the
-  layer [ix](https://ix.dev) publishes on top of its closed-source VM primitives.
-
-To explore, you could point Claude at this repo and ask whether anything here is
-useful for you.
-
-The clip above is not a screen recording. It is generated by
-[`reel`](packages/tui/reel/), which drives a real shell through the repo's own
-[PTY driver](packages/tui/tui/), rasterizes the styled grid with a flat palette, and
-encodes a 60fps animated AVIF (with a WebP fallback). AV1's inter-frame
-compression keeps it around 140 KB. Regenerate it any time:
-
-```sh
-nix run .#reel          # writes docs/demo-{dark,light}.{avif,webp}
-```
-
-## Quick Check
+It is one Nix flake holding ~45 packages (mostly Rust, with Python, Elixir,
+TypeScript, and Svelte where they fit), a corpus of NixOS modules and OCI
+images, and the agent infrastructure that ties them together. Most packages have
+a from-source page under [`docs/`](docs/index.md). To explore, point Claude at
+this repo and ask whether anything here is useful for you.
+
+## What's inside
+
+### Agent infrastructure
+
+The harness, governance, and tuning loop that runs coding agents (Claude Code and
+Codex) across the fleet under one set of rules. The house system prompt is
+[`system-prompt.nix`](packages/agent/system-prompt.nix), an ordered set of named,
+reviewable bindings rather than a text blob, so behavior changes land as PR diffs.
+
+| Package | What it does |
+| --- | --- |
+| [`claude-code`](packages/agent/claude-code/) / [`codex`](packages/agent/codex/) | Agent CLIs wrapped with the shared house prompt, MCP servers, and hooks baked in |
+| [`policy`](packages/agent/policy/) | One source of tool-access rules for both wrappers (deny force-merge, block superseded builtins) |
+| [`system-prompt-eval`](packages/agent/system-prompt-eval/) | Spawns sandboxed `claude -p` rollouts, scores them with an LLM judge, commits the scores |
+| [`claude-hooks`](docs/claude-hooks/overview.md) | Lifecycle hooks as one Rust binary; every hook fails open and silent, so a broken hook never blocks a session |
+| [`subagent-cache`](packages/agent/subagent-cache/) | Memoizes read-only investigations across the team, validated by Postgres recall + a Haiku judge + file-freshness hashing |
+| [`symphony`](packages/agent/symphony/) | Elixir/OTP runtime orchestrating multi-repo Codex sessions from a `.sym` DSL, each in its own git worktree |
+| [`distiller`](packages/agent/distiller/) | Turns raw session transcripts into searchable, reusable lessons |
+| [`pi-harnesses`](packages/agent/pi-harnesses/) | Fixed agent postures: sandboxed engine, beam-search executor, skeptical prosecutor |
+| [`claude-stories`](packages/agent/claude-stories/) | A status-line row of teammate avatars, peer-discovered over Tailscale with no central server |
+
+### A Nix build system rebuilt for speed
+
+| Package | What it does |
+| --- | --- |
+| [`nix-cargo-unit`](packages/nix/nix-cargo-unit/) | Renders the Cargo workspace as one content-addressed derivation per rustc unit, not per crate |
+| [`snix`](packages/nix/snix/) | A Rust reimplementation of Nix, built here with cargo-unit (~1100 crate builds collapse into one incremental graph) |
+| [`nix-fast-build`](packages/nix/nix-fast-build/) + [`nix-eval-jobs`](packages/nix/nix-eval-jobs/) | Patched to correctly skip already-realized CA outputs (an ~85s cache-check floor for ~1450 units becomes ~0.1s) |
+| [`oci-image-builder`](packages/nix/oci-image-builder/) | Splits image "describe" from "materialize" and shards per-layer tarring, so rebuilds stay fast and deterministic |
+| [`nix-web-monitor`](packages/nix/nix-web-monitor/) | Streams Nix's internal JSON build log into a live browser dashboard while the build runs in your terminal |
+| [`blast-radius`](packages/blast-radius/) | Reports how many derivations a PR would rebuild, and why |
+| [`indexbench`](packages/indexbench/) | Gates macro-benchmark and allocation-count regressions in CI |
+
+### Code intelligence and search
+
+| Package | What it does |
+| --- | --- |
+| [`search`](packages/search/search/) | Semantic code search by meaning; content-addressed, so identical files across branches share one embedding |
+| [`astlog`](packages/code/astlog/) | Datalog over tree-sitter syntax trees (matches as relations, joins as rules); gates `nix run .#lint` |
+| [`scipql`](packages/code/scipql/) | The same idea over a SCIP semantic index, so a rename never touches an unrelated same-named symbol |
+
+### Terminal automation
+
+| Package | What it does |
+| --- | --- |
+| [`tui`](packages/tui/tui/) | A PTY driver: drive any interactive program (gdb, vim, REPLs) and read back a rendered screen, not raw escape codes. Python + Node bindings |
+| [`reel`](packages/tui/reel/) | Records a terminal demo through the PTY driver and encodes it to animated AVIF/WebP (see below) |
+| [`run`](packages/tui/run/) | Records a command under a terminal session, keeping agent logs small |
+| [`dashboard`](packages/dashboard/) | A live grid of running terminals in the browser over a Loro CRDT and SSE |
+
+The demo at the top of this README is not a screen recording. [`reel`](packages/tui/reel/)
+drives a real shell through the PTY driver, rasterizes the styled grid with a flat
+palette and an embedded monospace face, and encodes a 60fps animated AVIF (WebP
+fallback). AV1's inter-frame compression keeps it around 140 KB. Regenerate it any
+time with `nix run .#reel`.
+
+### Agent-facing primitives
+
+A Python [`mcp`](packages/mcp/) server hands all of the above to an LLM with no
+install step. Its one general `python_exec` tool runs on a single shared,
+persistent IPython kernel: namespace persists across calls, work can background
+past the foreground budget, and sessions checkpoint to disk. Bundled modules
+expose search, the PTY driver, a `fleet` cluster API (Ray, Spark, SSH fan-out to
+Polars frames), browser and screen control, and cloud integrations.
+
+### VMs, images, and fleet
+
+The layer [ix](https://ix.dev) publishes on top of its closed-source VM primitives:
+ready-to-run [OCI images](images/) and reusable, auto-discovered [NixOS modules](modules/).
+
+| Package | What it does |
+| --- | --- |
+| [`vmkit`](packages/vm/vmkit/) | Spawns guests on macOS Virtualization.framework or Linux libkrun from one binary |
+| [`chrome-vm`](packages/vm/chrome-vm/) | Runs headless Chromium inside a real VM |
+| [`ix-fleet`](packages/ix-fleet/) | Drives declarative multi-VM rollouts |
+| [`dag-runner`](packages/dag-runner/) | Executes JSON task DAGs for parallel health checks |
+
+## Quick check
 
 ```sh
 nix flake show          # list every package, module, and check
 nix run .#lint          # nixfmt, statix, deadnix, astlog (nix + rust)
 nix build .#minecraft   # realize one image closure
+nix run .#reel          # regenerate the demo above
 ```
 
 ## Layout
 
-- [`packages/`](packages/) repo-owned tools (search, PTY driver, agent loops, MCP server, the `reel` demo recorder).
-- [`images/`](images/) runnable NixOS systems packaged as OCI archives.
-- [`modules/`](modules/) opt-in NixOS service modules, auto-discovered.
-- [`lib/`](lib/) shared helper and builder API.
-- [`examples/`](examples/) standalone consumer fleets.
-- [`skills/`](skills/) Claude Code skills, auto-discovered and shipped to agents.
+| Path | Contents |
+| --- | --- |
+| [`packages/`](packages/) | Repo-owned tools: agent stack, Nix build system, search, PTY driver, MCP server, `reel` |
+| [`images/`](images/) | Runnable NixOS systems packaged as OCI archives |
+| [`modules/`](modules/) | Opt-in NixOS service modules and profiles, auto-discovered |
+| [`lib/`](lib/) | Shared helper and builder API (Rust workspace graph, `buildUvApplication`, Minecraft/NBT, agent integration) |
+| [`docs/`](docs/index.md) | From-source documentation, one page per package |
+| [`examples/`](examples/) | Standalone consumer fleets |
+| [`skills/`](skills/) + [`agents/`](agents/) | Claude Code skills and subagent definitions, shipped to agents |
+| [`rfcs/`](rfcs/) | Architecture decision records |
 
 ## Feedback
 
 Bug reports and enhancement requests go to [GitHub Issues](https://github.com/indexable-inc/index/issues). Security reports follow [SECURITY.md](SECURITY.md). Code changes land through pull requests against the `main` branch; see [CONTRIBUTING.md](CONTRIBUTING.md) for local setup, coding standards, and commit conventions.
 
-## Contributor Notes
+## Contributor notes
 
 See [AGENTS.md](AGENTS.md) and [CONTRIBUTING.md](CONTRIBUTING.md) when you're ready to dig in.