|
| 1 | +# emitter-diff |
| 2 | + |
| 3 | +A language-agnostic tool for **diffing the generated code produced by two versions of a |
| 4 | +TypeSpec emitter**. |
| 5 | + |
| 6 | +It runs the emitter's own regenerate command against a **baseline** source tree and a **head** |
| 7 | +source tree, then shows the diff between the two generated outputs. Use it locally during |
| 8 | +development and in CI on PRs. |
| 9 | + |
| 10 | +The tool contains **zero language logic**. An emitter integrates by naming three things — its |
| 11 | +regenerate command, the package directory to run it in, and the generated-code directory to diff — |
| 12 | +either as flags or via a built-in `--emitter` preset. |
| 13 | + |
| 14 | +## How it works |
| 15 | + |
| 16 | +```text |
| 17 | + baseline tree ─► run <command> in <emitter-path> ─► snapshot <generated-code-path> ─┐ |
| 18 | + ├─► git diff ─► terminal / HTML |
| 19 | + head tree ─────► run <command> in <emitter-path> ─► snapshot <generated-code-path> ─┘ |
| 20 | +``` |
| 21 | + |
| 22 | +- A **source tree** is resolved for each side (`--baseline` / `--head`). |
| 23 | +- The **`--command`** is run verbatim (tokenized to argv — no shell) inside |
| 24 | + `<tree>/<emitter-path>`. This is the emitter's _unmodified_ regenerate command; the tool does |
| 25 | + not reach into it. |
| 26 | +- The `<emitter-path>/<generated-code-path>` subtree is snapshotted for each side and the |
| 27 | + two snapshots are diffed. `--generated-code-path` accepts **multiple roots** — an emitter that |
| 28 | + writes generated code to several directories (e.g. Go) passes a comma-separated list or repeats |
| 29 | + the flag; each root is snapshotted under its own relative path so outputs never collide. |
| 30 | + |
| 31 | +Because the tool just runs a command, **any emitter with a regenerate script works** — no per-language |
| 32 | +plugin code. |
| 33 | + |
| 34 | +## Usage |
| 35 | + |
| 36 | +```bash |
| 37 | +# Using a built-in preset (fills in command + paths): |
| 38 | +node eng/emitter-diff/src/cli.ts --emitter python --baseline gh:<sha> |
| 39 | + |
| 40 | +# Fully explicit (no preset needed): |
| 41 | +node eng/emitter-diff/src/cli.ts \ |
| 42 | + --command "npm run regenerate" \ |
| 43 | + --emitter-path packages/http-client-python \ |
| 44 | + --generated-code-path tests/generated \ |
| 45 | + --baseline gh:<sha> |
| 46 | + |
| 47 | +# Multiple generated roots (e.g. Go writes several) — comma-separated (or repeat the flag): |
| 48 | +node eng/emitter-diff/src/cli.ts \ |
| 49 | + --command "npm run tspcompile" \ |
| 50 | + --emitter-path packages/typespec-go \ |
| 51 | + --generated-code-path test/http-specs,test/azure-http-specs \ |
| 52 | + --baseline github:Azure/autorest.go@<sha> |
| 53 | +``` |
| 54 | + |
| 55 | +> This tool is a set of plain `.ts` scripts — not an installed package. It runs through `node` |
| 56 | +> (which executes TypeScript directly on the versions this repo supports), so there is nothing to |
| 57 | +> build. Typecheck with `npx tsc -p eng/emitter-diff`. |
| 58 | +
|
| 59 | +> **Build your checkout first.** `--head` defaults to the current working tree, and the tool runs |
| 60 | +> the regenerate command against it **as-is** — it never installs or builds your checkout (see |
| 61 | +> [Command prep](#command-prep---setup)). Build the emitter for the head side before diffing; for |
| 62 | +> **python** that's `npm run setup` in `packages/http-client-python` (builds the emitter and creates |
| 63 | +> the venv `regenerate` requires). Only trees the tool fetches from GitHub are auto-prepared. |
| 64 | +
|
| 65 | +### Emitter config |
| 66 | + |
| 67 | +| Flag | Meaning | |
| 68 | +| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |
| 69 | +| `--emitter <name>` | Built-in preset that fills in the three fields below. | |
| 70 | +| `--command <cmd>` | Regenerate command, run verbatim in `--emitter-path`. | |
| 71 | +| `--emitter-path <path>` | Package dir (relative to a tree root) to run the command in. | |
| 72 | +| `--generated-code-path <path>` | Generated-code dir (relative to `--emitter-path`) to diff. Accepts **multiple roots** — comma-separated (`a,b`) or by repeating the flag — for an emitter with several generated roots (e.g. Go); each is snapshotted under its own relative path. | |
| 73 | + |
| 74 | +A preset supplies all three; each flag still overrides the preset value. To onboard a new language, |
| 75 | +add a row to `EMITTER_DEFAULTS` in `src/registry.ts` — or just pass the three flags directly and |
| 76 | +skip `--emitter`. |
| 77 | + |
| 78 | +### Refs |
| 79 | + |
| 80 | +| Syntax | Meaning | |
| 81 | +| --------------------------------- | ------------------------------------ | |
| 82 | +| `local:/path` or `./path` | a local source folder (run in place) | |
| 83 | +| `github:owner/repo@<sha\|branch>` | a GitHub source at a ref | |
| 84 | +| `gh:<sha\|branch>` | this repo (origin remote) at a ref | |
| 85 | + |
| 86 | +`--head` defaults to the **current working tree**. `--baseline` defaults to the `upstream` remote's |
| 87 | +repo at its default branch, falling back to `origin` (e.g. `github:microsoft/typespec@main`). |
| 88 | + |
| 89 | +### Command prep (`--setup`) |
| 90 | + |
| 91 | +The `--command` is run **as-is** — it does not install deps or build. Instead, prep is handled by |
| 92 | +**`--setup`**, which runs **only in a tree the tool freshly fetched from GitHub** (a `gh:`/`github:` |
| 93 | +ref). The current working tree and user-provided `local:` paths are assumed already built and are |
| 94 | +**never touched** (so setup never mutates your checkout or a prepared CI worktree). |
| 95 | + |
| 96 | +- A **preset** supplies sensible setup defaults, so `--emitter python --baseline gh:main` installs + |
| 97 | + builds the fetched baseline automatically (python: `npm install --ignore-scripts` → `npm run setup`; |
| 98 | + typescript: `pnpm install` → `npm run build`). |
| 99 | +- Override with one or more `--setup <cmd>` (each runs in order, in `<tree>/<emitter-path>`), or |
| 100 | + disable entirely with `--no-setup`. |
| 101 | + |
| 102 | +Each `--setup` command — like `--command` — is tokenized to argv and run **without a shell**, so |
| 103 | +multi-step pipelines (`&&`, `|`) are not supported; pass repeated `--setup` flags instead. |
| 104 | + |
| 105 | +### Common options |
| 106 | + |
| 107 | +By default the tool writes a **clickable HTML report** (`emitter-diff.html`) into the work dir and |
| 108 | +prints a `file://` link to it. |
| 109 | + |
| 110 | +- `--baseline <ref>` / `--head <ref>`: the two source trees to compare. |
| 111 | +- `--setup <cmd>` (repeatable) / `--no-setup`: prep commands for freshly fetched GitHub trees. |
| 112 | +- `--work-dir <dir>`: scratch dir for snapshots (default: a fresh temp dir). |
| 113 | +- `--sequential`: regenerate baseline then head one after another instead of in parallel. Useful on |
| 114 | + a single machine where running both at once oversubscribes the CPU (each regenerate already fans |
| 115 | + out across cores) or trips generator races. |
| 116 | +- `--ci`: disable the local baseline-output cache (intended for CI). |
| 117 | +- `--html <file>`: write the rendered HTML report to this path. |
| 118 | +- `--md <file>`: write a Markdown report (collapsible per-file `diff` blocks) to this path. Handy for |
| 119 | + a CI job summary (`$GITHUB_STEP_SUMMARY`) or a PR comment body. |
| 120 | +- `--no-group`: render every file separately in the HTML/Markdown reports. By default, files that |
| 121 | + share the same change — the same added/removed lines, even when the surrounding generated code |
| 122 | + differs — are merged into a single collapsible group (with the shared diff shown once and every |
| 123 | + once and every affected file listed), so a repeated change across many generated files is reviewed |
| 124 | + once instead of N times. |
| 125 | +- `--fail-on-diff`: exit non-zero when output differs (exit `2` = diff present, `1` = hard error). |
| 126 | +- `-- <args>`: everything after `--` is appended to `--command` verbatim on **both** sides, so the |
| 127 | + diff stays apples-to-apples. Use it to regenerate only a subset of tests by forwarding the |
| 128 | + regenerate script's own filter flags. |
| 129 | + |
| 130 | +### Regenerating a subset of tests |
| 131 | + |
| 132 | +The tool doesn't define its own test-filter flags — it forwards `-- <args>` to each emitter's |
| 133 | +regenerate command, which owns the filtering. For the **Python** emitter (`regenerate.ts`): |
| 134 | + |
| 135 | +- `-n, --name <pattern>`: case-insensitive substring match on package name. |
| 136 | +- `-f, --flavor <azure|unbranded>`: limit to one flavor. |
| 137 | +- `-j, --jobs <n>`: parallel job count. |
| 138 | + |
| 139 | +```sh |
| 140 | +# Only the authentication packages, azure flavor |
| 141 | +node ../../eng/emitter-diff/src/cli.ts --emitter python -- --name authentication --flavor azure |
| 142 | + |
| 143 | +# Via the package script (first `--` is npm's, second is emitter-diff's passthrough separator) |
| 144 | +npm run diff-spector-tests -- -- --name type/array |
| 145 | +``` |
| 146 | + |
| 147 | +`--name` filters the spec set already bundled in the package's `node_modules` |
| 148 | +(`@azure-tools/azure-http-specs` + `@typespec/http-specs`); it does not point the test set at an |
| 149 | +arbitrary folder or ref. Other emitters expose their own filter flags — pass whatever their |
| 150 | +regenerate command accepts. |
| 151 | + |
| 152 | +## CI integration |
| 153 | + |
| 154 | +`.github/workflows/ci-emitter-diff-<lang>.yml` runs on PRs that touch the language emitter or this |
| 155 | +tool. The **baseline** is the base-branch commit the PR is based on (the `git merge-base` with the |
| 156 | +target branch). Because the tool runs the regenerate command as-is, the workflow **prepares both |
| 157 | +trees** (installs deps, builds the emitter, creates any venv) before invoking the tool, then: |
| 158 | + |
| 159 | +- posts a **sticky PR comment** (updated in place on each push) linking the diff artifact, and |
| 160 | +- uploads the rendered **HTML report** as an artifact. |
| 161 | + |
| 162 | +**Informational:** the check **always passes unless the tool hits a real tool/build error** — a |
| 163 | +generated-output diff does not fail the PR. CI runs the tool without `--fail-on-diff`, so a diff |
| 164 | +still exits `0`; only a non-zero exit (a build/venv/generate failure) fails the job. |
| 165 | + |
| 166 | +**Fork PRs are not run.** The job checks out and executes the PR's code (builds the emitter, runs |
| 167 | +`regenerate`), so a job-level `if` guard restricts it to same-repo PRs — it skips any PR whose head |
| 168 | +is a fork. |
| 169 | + |
| 170 | +## Adding a new language |
| 171 | + |
| 172 | +Either add a preset row to `EMITTER_DEFAULTS` (`src/registry.ts`): |
| 173 | + |
| 174 | +```ts |
| 175 | +rust: { |
| 176 | + command: "npm run tspcompile", |
| 177 | + emitterPath: "packages/typespec-rust", |
| 178 | + generatedCodePath: "test/generated", |
| 179 | +}, |
| 180 | +``` |
| 181 | + |
| 182 | +…or skip the preset entirely and pass `--command` / `--emitter-path` / |
| 183 | +`--generated-code-path` directly. Either way, ensure each side's tree can actually run the |
| 184 | +command (see **Command prep** above). The orchestrator, ref resolver, and diff engine need no |
| 185 | +changes. |
| 186 | + |
| 187 | +## Notes & limitations |
| 188 | + |
| 189 | +- `--html` renders a self-contained, GitHub-style HTML report (inline CSS, no external requests). |
| 190 | + `--md` renders a Markdown report (collapsible per-file `diff` blocks) suitable for a CI job |
| 191 | + summary or PR comment. Both reports group files that share the same change into one block by |
| 192 | + default (`--no-group` to disable), so reviewing a repeated diff is a one-time effort. The diff |
| 193 | + itself is produced by `git diff --no-index`; the tool leans on only a couple of small repo dev |
| 194 | + dependencies (`execa`, `picocolors`) for process spawning and terminal coloring. |
| 195 | +- For github refs (including fork repos), the resolver uses a detached, commit-keyed cached git |
| 196 | + worktree under the temp directory, created from an isolated cache repo (not from your active |
| 197 | + checkout). Repeated runs on the same commit reuse it. |
| 198 | +- Spec inputs come from each side's own dependencies (e.g. `node_modules/@typespec/http-specs`). |
| 199 | + Spec versions rarely change, so any drift between baseline/head is treated as acceptable noise. |
0 commit comments