Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 36 additions & 91 deletions packages/graph/README.md

Large diffs are not rendered by default.

24 changes: 12 additions & 12 deletions website/src/content/blog/i-made-ts-compiler-graph-mcp.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -195,11 +195,11 @@ I came to the same problem holding two things they didn't: a compiler toolchain

| Pain (§2) | Antidote (§3) |
| --- | --- |
| 2.4over-forced (up to a hard ban), or no guidance at all | **3.1** guides without forcing (`escape` + stop) |
| 2.3a real graph buried under 14 (or ~50) tools | **3.2** one tool, asked in plain language |
| 2.3source bodies blow up tokens | **3.3** returns `index` only |
| 2.3a setup tax before the tools even work | **3.4** a free byproduct, no setup step |
| (only works if trustworthy) | **3.4** the real compiler |
| 2.4: over-forced (up to a hard ban), or no guidance at all | **3.1**: guides without forcing (`escape` + stop) |
| 2.3: a real graph buried under 14 (or ~50) tools | **3.2**: one tool, asked in plain language |
| 2.3: source bodies blow up tokens | **3.3**: returns `index` only |
| 2.3: a setup tax before the tools even work | **3.4**: a free byproduct, no setup step |
| (only works if trustworthy) | **3.4**: the real compiler |

### 3.1. It Never Forces You

Expand Down Expand Up @@ -282,13 +282,13 @@ This is also the answer to the hand-shaping problem. You just ask in plain Engli

And every fork in that reasoning is a discriminated union. The `request` is one, tagged by a `type` string: `ITtscGraphLookup.IRequest` is `type: "lookup"`, `ITtscGraphTour.IRequest` is `type: "tour"`, and so on, so choosing the literal _is_ the decision of which operation to run. The chain-of-thought walks that fork in stages: `draft.type` names the branch it intends, `review` is its chance to switch to a different one, and the committed `request` is the branch it lands on. The returned `result.type` comes back on the same tag, so the agent always knows which shape it's holding. Each branch is one graph operation with its own request (`.IRequest`) and result type:

- [`ITtscGraphTour`](https://github.com/samchon/ttsc/blob/master/packages/graph/src/structures/ITtscGraphTour.ts) a broad code tour. Answers "new project, how does it work?" in one index.
- [`ITtscGraphEntrypoints`](https://github.com/samchon/ttsc/blob/master/packages/graph/src/structures/ITtscGraphEntrypoints.ts) find the entry points. Where to start reading.
- [`ITtscGraphLookup`](https://github.com/samchon/ttsc/blob/master/packages/graph/src/structures/ITtscGraphLookup.ts) find a symbol by name.
- [`ITtscGraphTrace`](https://github.com/samchon/ttsc/blob/master/packages/graph/src/structures/ITtscGraphTrace.ts) trace call / data flow (forward, or impact radius).
- [`ITtscGraphDetails`](https://github.com/samchon/ttsc/blob/master/packages/graph/src/structures/ITtscGraphDetails.ts) a chosen symbol's signature, members, neighbors.
- [`ITtscGraphOverview`](https://github.com/samchon/ttsc/blob/master/packages/graph/src/structures/ITtscGraphOverview.ts) a repo-level overview.
- [`ITtscGraphEscape`](https://github.com/samchon/ttsc/blob/master/packages/graph/src/structures/ITtscGraphEscape.ts) a no-op bail-out, out of the graph.
- [`ITtscGraphTour`](https://github.com/samchon/ttsc/blob/master/packages/graph/src/structures/ITtscGraphTour.ts): a broad code tour. Answers "new project, how does it work?" in one index.
- [`ITtscGraphEntrypoints`](https://github.com/samchon/ttsc/blob/master/packages/graph/src/structures/ITtscGraphEntrypoints.ts): find the entry points. Where to start reading.
- [`ITtscGraphLookup`](https://github.com/samchon/ttsc/blob/master/packages/graph/src/structures/ITtscGraphLookup.ts): find a symbol by name.
- [`ITtscGraphTrace`](https://github.com/samchon/ttsc/blob/master/packages/graph/src/structures/ITtscGraphTrace.ts): trace call / data flow (forward, or impact radius).
- [`ITtscGraphDetails`](https://github.com/samchon/ttsc/blob/master/packages/graph/src/structures/ITtscGraphDetails.ts): a chosen symbol's signature, members, neighbors.
- [`ITtscGraphOverview`](https://github.com/samchon/ttsc/blob/master/packages/graph/src/structures/ITtscGraphOverview.ts): a repo-level overview.
- [`ITtscGraphEscape`](https://github.com/samchon/ttsc/blob/master/packages/graph/src/structures/ITtscGraphEscape.ts): a no-op bail-out, out of the graph.

### 3.3. Index Only, So Nothing Explodes

Expand Down
66 changes: 8 additions & 58 deletions website/src/content/docs/benchmark/graph.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,71 +2,21 @@ import GraphBenchmark from "../../../components/benchmark/GraphBenchmark";

# Code Graph (MCP)

`@ttsc/graph` is the subject of this benchmark page. The chart measures agent cost with and without the graph, then compares that result with the two external comparator arms used by the benchmark harness. Bar labels such as `486k tokens` are token usage, and graph or comparator rows also show the token reduction versus the baseline row for that case.
This benchmark measures what an agent spends to answer a real question, with and without `@ttsc/graph`, against three comparators on the same prompts: `codegraph`, `codebase-memory-mcp`, and serena. Bar labels such as `486k tokens` are token usage, and graph or comparator rows also show the token reduction versus the no-MCP baseline for that case.

<GraphBenchmark />

## What `@ttsc/graph` Exposes
## How to read it

`packages/graph/src/structures/ITtscGraphApplication.ts` defines one MCP method, `inspect_typescript_graph`. The request wrapper forces the agent to fill `question`, `draft`, `review`, and exactly one request branch before it calls the tool.
The chart runs two prompt families, a shared onboarding question and `codegraph`'s own per-repo questions, across eight real repositories and four models. Switch models and repositories with the tabs.

The response wrapper is also fixed: the method returns `{ result: ... }`, and `result` is the selected branch.
`@ttsc/graph` holds a flat, low median token cost as the repository grows. The comparators swing with repository size, and on some cases land above the no-MCP baseline, so the agent spends more with the tool than without it. This is one person's benchmark over a bounded set of scenarios: the [launch post](/blog/i-made-ts-compiler-graph-mcp) covers the method in full and where the numbers hold or break down, and the [Comparison](/docs/graph/compare) page explains the design reasons behind the gap.

| Named type | Role |
| --- | --- |
| `ITtscGraphApplication.IProps` | The request wrapper with `question`, `draft`, `review`, and `request`. |
| `ITtscGraphApplication.IResult` | The result wrapper. `result` is the selected branch. |
| `ITtscGraphNext` | The follow-up contract. `next.action` is `answer`, `inspect`, `outside`, or `clarify`. |
| `ITtscGraphTour` | The answer-ready repository-orientation branch. |
| `ITtscGraphEntrypoints` | The first-pass shortlist branch. |
| `ITtscGraphLookup` | The concrete symbol lookup branch. |
| `ITtscGraphTrace` | The flow branch for callers, callees, or dependency paths. |
| `ITtscGraphDetails` | The selected-handle branch with signatures, members, direct relationships, and sourceSpan anchors. |
| `ITtscGraphOverview` | The broad architecture branch. |
| `ITtscGraphEscape` | The no-op branch for evidence that belongs outside the indexed TypeScript graph. |
## What is compared

## Request And Result Contract
`@ttsc/graph` exposes one typed MCP tool. Its request and result contract, with the live type sources, is on the [Design](/docs/graph/design) page. The three comparators run from their own published entry points and stay separate arms rather than folding into one story, since each resolves and returns a graph differently. The [Comparison](/docs/graph/compare) page lays those differences out.

The server guidance in `packages/graph/src/server/instructions.ts` and the result guidance in `packages/graph/src/server/resultGuide.ts` say the same thing in different places: ask for the smallest graph evidence that can answer, stop when `next.action` is `answer`, and use `escape` when the next evidence is outside the graph.

| Request | What it is for | Result evidence |
| --- | --- | --- |
| `tour` | Repository orientation, code-tour, read-next, and broad runtime-flow questions. | Central entrypoints, primary flow, nearby paths, test anchors, and answer anchors. |
| `entrypoints` | First pass on a natural-language code question. | Ranked symbols, direct mentions, and a small orientation slice. |
| `lookup` | Resolve a concrete name or dotted handle. | Ranked hits with signatures when available. |
| `trace` | Follow runtime or type flow, or answer "how A reaches B". | Hops, reached nodes, `steps`, and `path` when `to` is set. |
| `details` | Inspect one to three chosen handles. | Signatures, member outlines, direct relationships, and sourceSpan anchors. |
| `overview` | Broad project orientation. | Counts, layers, hotspots, and public API symbols. |
| `escape` | Stop when the next evidence is outside the graph. | `skipped: true` plus a reason and next step. |

The payloads are coordinates and relationships, not source bodies. `sourceSpan` and `evidence` point at file and line ranges, and `next` tells the caller whether to answer, inspect once more, leave graph, or clarify.

## Source Contract

The benchmark page keeps source text out of the first render so the chart remains responsive. The contract lives in these package files, and the full type/source view is on the [graph tools page](/docs/graph/tools/).

| File | Role |
| --- | --- |
| `packages/graph/src/structures/ITtscGraphApplication.ts` | MCP request/result wrapper. |
| `packages/graph/src/structures/ITtscGraphTour.ts` | Repository-orientation result branch. |
| `packages/graph/src/structures/ITtscGraphNext.ts` | Follow-up action contract. |
| `packages/graph/src/server/instructions.ts` | Tool instruction text. |
| `packages/graph/src/server/resultGuide.ts` | Per-result answer guidance. |

## How The Comparators Differ

The benchmark harness treats `codegraph`, `codebase-memory-mcp`, and Serena as separate comparator arms, not as the same contract.

| Tool | Local setup in the benchmark source | What this page treats it as |
| --- | --- | --- |
| `@ttsc/graph` | The typed MCP server defined in `packages/graph/src/structures` and `packages/graph/src/server`. | The subject of the page. |
| `codegraph` | `codegraph init`, with `.codegraph/` ignored and optionally kept for inspection. | External comparator arm. |
| `codebase-memory-mcp` | `codebase-memory-mcp cli index_repository`, with an isolated `CBM_CACHE_DIR` and `.codebase-memory/` ignored. | External comparator arm. |
| Serena | `uvx --from git+https://github.com/oraios/serena serena start-mcp-server --context <client> --project <repo>`, with `.serena/` ignored and optionally kept for inspection. | External comparator arm. |

The chart reads raw samples from `website/public/benchmark/graph.json`, so it keeps those setup paths separate instead of folding them into one story.

## Benchmark Your Project
## Reproduce it

Start from a clean checkout, install the workspace, build `@ttsc/graph`, then run the same prompt once without MCP and once with the graph arm.

Expand All @@ -80,4 +30,4 @@ node experimental/benchmark/graph/agent-ab-codex.mjs --repo=typeorm --repo-dir=/
node experimental/benchmark/graph/agent-ab-codex.mjs --repo=typeorm --repo-dir=/abs/path/to/project --tsconfig=tsconfig.json --question "Trace the main runtime path I should understand before editing." --arm=graph --runs=1
```

Use `experimental/benchmark/README.md` for the full matrix and publication flow.
The chart reads raw samples from `website/public/benchmark/graph.json`. For the full model matrix, the comparator setup commands, and the publication flow, see `experimental/benchmark/README.md`.
3 changes: 2 additions & 1 deletion website/src/content/docs/graph/_meta.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@ import type { MetaRecord } from "nextra";

const meta: MetaRecord = {
index: "Overview",
tools: "Tools",
design: "Design",
compare: "Comparison",
viewer: "3D Viewer",
};
export default meta;
Loading
Loading