feat: SEO, AI discoverability, and OG image generation

- public/robots.txt: adopt Content Signals policy (allow all bots; explicit
  GPTBot, Claude-Web, PerplexityBot, GoogleBot directives)
- src/routes/llms[.]txt.ts: hand-crafted Cadence explainer for LLM context
  windows covering resources, transaction model, capabilities, cross-VM,
  randomness, and curated links
- src/routes/llms-full[.]txt.ts: auto-generated full corpus via Fumadocs
  source loader (JSX-stripped, sorted by URL)
- src/lib/derive-description.ts: extract first prose paragraph for meta
  descriptions when frontmatter description is absent
- src/routes/og.docs.$.tsx + og.home.tsx: server-side OG images via
  @vercel/og / Satori using PNG base64 icons (SVG paths unsupported)
- AGENTS.md: updated with stack overview, gotchas, and tools section for
  coding agents (flow-ai-tools plugin + flow mcp MCP server)

Author: Aliserag

AGENTS.md

@@ -4,56 +4,117 @@ Guidance for AI coding agents working in this repository. Loaded into agent cont
 
 ## Overview
 
-`cadence-lang.org` is the Docusaurus v3 documentation site for the Cadence smart contract programming language (Flow network). Content lives in `docs/` as Markdown/MDX, rendered by Docusaurus with Code Hike for syntax-highlighted code blocks and an optional Typesense search index. Deployed at https://cadence-lang.org.
+`cadence-lang.org` is the documentation and marketing site for the Cadence smart contract programming language (Flow network). Built with TanStack Start (file-based routing + Nitro SSR) + Fumadocs (MDX docs engine) + Tailwind CSS v4 + Bun. Deployed on Vercel. Includes AI chat (`/api/chat`), Orama-powered search (federated with onflow/docs via the `external/onflow-docs` submodule), and `/llms.txt` + `/llms-full.txt` LLM-optimized endpoints. The Cadence MCP server is built into the Flow CLI as `flow mcp` ([source](https://github.com/onflow/flow-cli/tree/master/internal/mcp)), not bundled here.
 
 ## Build and Test Commands
 
-Node 20.x required (`package.json` `engines.node`).
-
-- `npm i` — install dependencies
-- `npm run start` — local dev server at http://localhost:3000
-- `npm run build` — production build into `build/` (used by CI)
-- `npm run serve` — serve the built site
-- `npm run clear` — clear Docusaurus caches (`.docusaurus/`, `.cache-loader/`)
-- `npm run swizzle` — swizzle a Docusaurus theme component
-- `npm run write-translations` / `npm run write-heading-ids` — Docusaurus i18n / heading-id maintenance
-- `npm run deploy` — `docusaurus deploy` (GitHub Pages); production deploy is actually driven by Vercel
-
-No test suite is configured — there are no test scripts in `package.json`. CI (`.github/workflows/deploy.yml`) validates changes by running `npm run build` on push to `main`.
-
-## Architecture
-
-- `docusaurus.config.js` — single source of site config: navbar, `onBrokenLinks: "throw"`, versioned docs (`current` labeled "1.0"), JSON-LD `headTags` (Organization, WebSite, ComputerLanguage, FAQPage), conditional Typesense theme, Code Hike remark plugin (Nord theme, line numbers, copy button), optional `GTAG` analytics, `scripts: ['/hotjar.js']`.
-- `sidebars.js` — one autogenerated sidebar (`docSidebar`) over the entire `docs/` tree; ordering is driven by `_category_.json` files inside each docs subfolder.
-- `docs/` — all documentation content:
-  - `language/` — Cadence language reference; subdirs `accounts/`, `operators/`, `types-and-type-system/`, `values-and-types/`
-  - `tutorial/` — numbered tutorial pages (`01-first-steps.md` through `09-voting.md`)
-  - `cadence-migration-guide/` — Cadence 1.0 migration content
-  - top-level pages: anti-patterns, design-patterns, security-best-practices, solidity-to-cadence, testing-framework, contract-upgrades, json-cadence-spec, project-development-tips, measuring-time, why, index
-- `src/` — site code (React):
-  - `pages/` — custom routes (`index.js`, `community.js`) plus JSON content files consumed by them
-  - `components/` — `HomepageFeatures/`, `Details/`, `feedbackFaces.tsx`
-  - `theme/` — swizzled Docusaurus components: `Admonition/`, `AnnouncementBar/`, `Details/`, `DocCard/`, `Layout/`, `TOC/`, `MDXComponents.js`, `NotFound.js`
-  - `ui/design-system/`, `css/custom.css`, `utils/gtags.client.ts`
-- `static/` — copied verbatim to build root: `favicon.ico`, `fonts/`, `img/`, `llms.txt`, `robots.txt`, `hotjar.js` (loaded via the `scripts` field in `docusaurus.config.js`)
-- `versions.json` — `["current"]`; only the live version exists, labeled "1.0" in config. No archived versions — don't run `docusaurus docs:version` unless intentionally branching.
-- `vercel.json` — URL rewrites/redirects (e.g. legacy `/docs/1.0/*` → `/docs/*`, `/docs/language/<type-page>` → `/docs/language/types-and-type-system/<type-page>`). Update here when moving pages.
-- `.github/workflows/deploy.yml` — on push to `main`, build check + Typesense scrape using `docsearch.config.json`. `scrape.yml` also exists in the same folder.
+Bun ≥ 1.0 required. Node ≥ 22 (Vercel uses 24.x). Flow CLI ≥ v2.16.0 for `flow mcp`.
+
+- `bun install` — install dependencies (run after pulling submodule changes too)
+- `bun run dev` — local dev server at http://localhost:3000
+- `bun run build` — production build into `.vercel/output/` + sitemap generation. `NODE_OPTIONS=--max-old-space-size=8192` is baked into the script (the SSR `source-*.mjs` chunk is ~13 MB and OOMs on default heap).
+- `bun run start` — serve the built server (`bun .output/server/index.mjs`)
+- `bun run types:check` — `fumadocs-mdx && tsc --noEmit`. May report errors against the `external/onflow-docs` submodule's own files; those are out-of-scope for this repo.
+- `git submodule update --init --recursive` — populate `external/onflow-docs` (required before search index can include onflow-docs content)
+
+No site-build CI in this repo — Vercel builds previews on every PR push.
+
+## Stack and Toolchain
+
+- **Bun is the toolchain.** Don't introduce npm/yarn lock files — `bun.lock` is the source of truth.
+- **TanStack Start** — file-based routing under `src/routes/`. Nitro adapter for SSR (Vercel preset). `src/routeTree.gen.ts` is auto-generated.
+- **Fumadocs** — `fumadocs-core`, `fumadocs-ui`, `fumadocs-mdx`. Source loader at `src/lib/source.ts`. MDX config in `source.config.ts`.
+- **Tailwind CSS v4** — via `@tailwindcss/vite` plugin (no config file). Theme tokens are CSS variables in `src/styles/app.css`. Use `cn()` from `src/lib/cn.ts` for class merging.
+- **`@vercel/og`** — server-side OG images via Satori + resvg-wasm.
+- **Dark mode** uses the `dark` class on `<html>` (Fumadocs convention).
 
 ## Conventions and Gotchas
 
-- **Node 20.x is pinned** (`package.json` `engines`; CI uses `actions/setup-node@v3` with `node-version: 20`). Don't bump one without the other.
-- **Docusaurus is pinned to 3.0.0** exactly (no caret). `@docusaurus/core`, `@docusaurus/preset-classic`, and `@docusaurus/module-type-aliases` must move together.
-- **Broken links fail the build.** `onBrokenLinks: "throw"` in `docusaurus.config.js`. When renaming a docs page, grep across `docs/` and `src/` and update references (or add a redirect in `vercel.json`).
-- **Typesense search is env-gated.** The theme only loads when both `TYPESENSE_NODE` and `TYPESENSE_SEARCH_ONLY_API_KEY` are set (see `.env.example`). Locally, search UI is absent unless those are set.
-- **Code Hike is active on all docs code blocks** (`remarkCodeHike` with Nord theme, `lineNumbers: true`, `showCopyButton: true`). Standard fenced code blocks work as-is.
-- **Sidebar is autogenerated.** Control labels and position via each folder's `_category_.json`, not by editing `sidebars.js`.
-- **Tailwind scans `src/**/*` and `docs/**/*.mdx`** only (`tailwind.config.js` `content`). Tailwind classes inside plain `.md` docs will not be generated — use `.mdx` if you need them.
-- **URL redirects belong in `vercel.json`**, not page front-matter. Follow existing patterns there when restructuring.
-- **Licensing is split:** source under Apache 2.0 (`LICENSE.txt`), content under CC-BY-4.0 (`CC-BY-4.0.txt`); see `LICENSE.md`.
-
-## Files Not to Modify
-
-- `package-lock.json` — regenerated by `npm i`
-- `/node_modules`, `/build`, `.docusaurus/`, `.cache-loader/` — gitignored build artifacts
-- `static/llms.txt` — curated for LLM discovery; verify source before editing
+These are the things a competent dev would otherwise get wrong.
+
+- **`NODE_OPTIONS=--max-old-space-size=8192` is required for builds.** Baked into `package.json`. The SSR `source-*.mjs` chunk grew past the 4 GB default heap when the `external/onflow-docs` submodule was added. Don't strip the flag.
+- **Submodule pin is deliberate, not branch-tracking.** `external/onflow-docs` points to a specific commit. Bumping it is a semi-supply-chain action — review the diff.
+- **Satori (the `@vercel/og` renderer) cannot render SVG `<path>` elements.** For per-doc OG images at `og.docs.$.tsx`, use PNG base64 data URIs via `src/lib/og-icon.ts`. Inline SVG paths will silently fail.
+- **URL redirects belong in `vercel.json`**, not page frontmatter. When moving or renaming a page, add the redirect there.
+- **Server-only handlers use the TanStack pattern:**
+
+```ts
+  export const Route = createFileRoute('/api/example')({
+    server: {
+      handlers: {
+        GET: async ({ request }) => { /* ... */ },
+      },
+    },
+  });
+```
+  
+  See `src/routes/api/chat.ts` and `src/routes/api/search.ts`. `ANTHROPIC_API_KEY` is server-only and never reaches the browser bundle.
+
+- **Section / page ordering is per-folder `meta.json`** (Fumadocs convention; replaces Docusaurus `_category_.json`).
+- **MDX frontmatter shape:**
+
+```yaml
+  ---
+  title: Page Title
+  description: One-line description for OG and search.
+  icon: optional-icon-name
+  slug: optional-url-override
+  ---
+```
+
+- **Cadence syntax highlighting** uses a custom TextMate grammar at `src/lib/cadence.tmLanguage.json`. Shiki dual themes (`github-light` / `github-dark`) are CSS-variable controlled in `src/styles/app.css`.
+- **Admonition types** (defined in `source.config.ts`): `note`, `tip`, `info`, `warn`, `warning`, `danger`, `important`, `success`. Don't invent new ones without updating the schema.
+- **`external/onflow-docs` is indexed for search but not rendered in the docs nav.** Cross-site search hits link out to `developers.flow.com`.
+- **Licensing is split:** source under Apache 2.0 (`LICENSE.txt`), content under CC-BY-4.0 (`CC-BY-4.0.txt`). See `LICENSE.md`.
+
+## Tools for Coding Agents
+
+If you are writing Cadence code in this repo, install these before starting:
+
+```bash
+# Claude Code plugin — 11 Cadence-specific skills (language, audit, scaffold, test, DeFi, etc.)
+/plugin marketplace add onflow/flow-ai-tools
+/plugin install flow-dev@flow-ai-tools
+/reload-plugins
+
+# Cadence MCP server — type checking, symbol lookup, on-chain queries (requires Flow CLI ≥ v2.16.0)
+claude mcp add --scope user cadence-mcp -- flow mcp
+```
+
+Or install both in one command:
+
+```bash
+sh -ci "$(curl -fsSL https://raw.githubusercontent.com/onflow/flow-ai-tools/main/scripts/install.sh)"
+```
+
+## AI Surfaces
+
+- **`/api/chat`** — Anthropic streaming via `@ai-sdk/anthropic`. Server-only key.
+- **Search** — Orama index over `content/docs/` + `external/onflow-docs/`. Panel at `src/components/search.tsx`, Cmd+/ hotkey, localStorage persistence.
+- **`/llms.txt`** / **`/llms-full.txt`** — TanStack server routes emitting LLM-optimized markdown.
+- **MCP server** — `flow mcp` (Flow CLI, ≥ v2.16.0). Tools: `cadence_check`, `cadence_hover`, `cadence_definition`, `cadence_symbols`, `cadence_completion`, `get_contract_source`, `get_contract_code`, `cadence_execute_script`. stdio transport. Source: [onflow/flow-cli/internal/mcp](https://github.com/onflow/flow-cli/tree/master/internal/mcp); maintained by Flow DX.
+
+## Permission Boundaries
+
+### ✅ Always
+- Run `bun run types:check` before committing.
+- Use `cn()` for class merging; don't string-concatenate Tailwind classes.
+- Add URL redirects to `vercel.json` when renaming or moving a page.
+
+### ⚠️ Ask first
+- Bumping the `external/onflow-docs` submodule pin (review the diff; it affects search corpus and `bun run build` memory).
+- Adding or removing admonition types in `source.config.ts`.
+- Adding a new top-level runtime dependency (`bun add ...`).
+- Modifying existing entries in `vercel.json` (vs. adding new ones).
+- Changing the `NODE_OPTIONS` build flag.
+
+### 🚫 Never
+- Commit `.env*` files or any secret. `ANTHROPIC_API_KEY` is server-only.
+- Edit `bun.lock` by hand — regenerate via `bun install`.
+- Edit `src/routeTree.gen.ts`, `.source/`, `.vercel/output/`, `node_modules/`, or `build/` — all auto-generated.
+- Edit files inside `external/onflow-docs/` directly. To change content there, modify the upstream `onflow/docs` repo and bump the submodule pin.
+- Introduce npm or yarn lock files.
+
+## Notes
+
+Programmatic checks specified here are advisory — agents may skip them. CI on Vercel enforces the build on PR.

public/robots.txt

@@ -0,0 +1,23 @@
+# cadence-lang.org robots.txt
+# algolia: 98E1096D4FD67E70
+#
+# Content Signals Policy (contentsignals.org, 2025) — explicitly permit
+# search indexing, live AI retrieval (ChatGPT / Claude / Perplexity / etc.
+# citation), and AI training on all content. Cadence docs are open source
+# and intended to be maximally discoverable by humans and AI agents.
+#
+# Content-Signal: search=yes, ai-input=yes, ai-train=yes
+
+User-agent: *
+# Don't waste crawl budget on JSON/binary endpoints (/api/* returns
+# JSON, /og/* renders OG images already linked from each page's meta).
+Disallow: /api/
+Disallow: /og/
+Allow: /
+
+Sitemap: https://cadence-lang.org/sitemap.xml
+
+# Machine-readable AI indexes (complementary, non-standard):
+#   Curated:   https://cadence-lang.org/llms.txt
+#   Corpus:    https://cadence-lang.org/llms-full.txt
+#   Companion: https://developers.flow.com/llms.txt

src/lib/derive-description.ts

@@ -0,0 +1,44 @@
+/**
+ * Extracts the first substantive prose paragraph from processed (JSX-stripped)
+ * markdown. Used to auto-generate meta descriptions and llms.txt annotations
+ * for pages that lack a frontmatter `description` field.
+ */
+export function deriveDescription(processedMarkdown: string): string {
+  const blocks = processedMarkdown.split(/\n{2,}/);
+  for (const block of blocks) {
+    const trimmed = block.trim();
+    if (!trimmed) continue;
+    // Skip headings, frontmatter fences, code blocks, JSX components, lists, blockquotes
+    if (
+      trimmed.startsWith('#') ||
+      trimmed.startsWith('---') ||
+      trimmed.startsWith('```') ||
+      trimmed.startsWith('<') ||
+      trimmed.startsWith('-') ||
+      trimmed.startsWith('*') ||
+      trimmed.startsWith('>') ||
+      /^\s*\d+\.\s/.test(trimmed) ||
+      trimmed.startsWith(':::')
+    ) {
+      continue;
+    }
+    // Skip processed-MDX heading text + anchor like "Introduction [#introduction]"
+    if (/^\S[^\n]{0,80}\[#[\w-]+\]\s*$/.test(trimmed)) continue;
+    // Strip markdown syntax: [text](url) → text, **x**/*x*/`x` → x, <Component/> → '',
+    // and inline anchor markers like [#some-id]
+    const plain = trimmed
+      .replace(/\[#[\w-]+\]/g, '')
+      .replace(/\[([^\]]+)\]\([^)]+\)/g, '$1')
+      .replace(/<[^>]+>/g, '')
+      .replace(/[*_`]/g, '')
+      .replace(/\s+/g, ' ')
+      .trim();
+    if (plain.length < 20) continue;
+    if (plain.length <= 160) return plain;
+    // Truncate on word boundary, append ellipsis
+    const cut = plain.slice(0, 157);
+    const lastSpace = cut.lastIndexOf(' ');
+    return (lastSpace > 100 ? cut.slice(0, lastSpace) : cut) + '…';
+  }
+  return '';
+}

src/lib/og-icon.ts

@@ -0,0 +1,2 @@
+// Cadence icon as base64 PNG for OG images (Satori can't render SVG paths)
+export const CADENCE_ICON_DATA_URI = 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAAAXNSR0IArs4c6QAAAERlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA6ABAAMAAAABAAEAAKACAAQAAAABAAAAMKADAAQAAAABAAAAMAAAAADbN2wMAAAFI0lEQVRoBe1ZXWgcVRT+7mw2u0mplvrTpP5gzcZatfhSFfFBgg+CgoLQgvjSB7H1xYdWtAbTTmKfqkbBNlS0RkRBjFqwPvhSi74J4oskMSYloJjGqE2an2bT3czxu2lmMzM7szOzOyY+5MLs3L9z7nd+7jl37gLrZV0DNWlA1UTtJT5n1uE21Hm7E2kXNws+uViAaVpOfskJ8Ju5FQt4lsybgOTYroAVCwa+xWTzGezaV7D7k9HWLy9txBX1EUQeJnbyFJt/sm/Bdmy98D2Z/m0zNuxK1e+B9makGj8n+Eeugq+aUwRClUU+nXJOrE2An9u3oD7zJl2mzcl0NevVu9C5vVlkM+/SWx6jy6RXE7RzreoEGDKvh6FOEviTTmZrUY8vgHyWwujgg1hEnq7TlzhokSyUup/K2RKFd3wBsNtCtvMs5vBdlAViz8lkUigsHKJrvkja0D0aXwCldIy8HBtYHILhznFOX6QVQgUInRBn3cTmKmpeJFI2/H8KEEMTlV1o1MxiJt8AXBvA8hKwMTuPbSY3dJKF2o+kf1Q4eI2am+iFB9CQvRcoHT3cKKXhHxTlDXYOuAd8Wv3mZlxTR2Usl0VlQQqTvsKL0tmWIoQfSfwtILtTGAEzLJ7hkwFcB8BlBGphaXyBM8PK+a77YMk7yBdvcEydZi7Rkeasoy92tVwArflhvEX59wZyU5jg2H60HjkdOMceGDHvgWX1sLnL7rr6VpPUS6O7z9HSLhRuAE+cHX99A93mVSaSpx2svNUJMj6IlqavvQNl7V87HmBfLx8PeD2T6AwjACJdKGIUcltgbvY6iHqCzOk2fkXNI8Uz/8Wmb6BWzuR+M6HBq9QHxHmX73ilTrUcQgPEc5K6BTCUQQukAyIA2clzuN0842TgW9duAxwPAT8HJQxjtZXoeUCpPH32x9DlbLcRP7exqen/SnXhsvWD3eN6628vUZECqdsCLi6ehminDSnR3GaKVt6H7Tu+hNrD44JP0ZmYEviMlHVFt0AZqadDu42ROs7eSj5PzRsv44KcDgTvYRvWjG6BSpyGTQ26lzbyiTYlwikq9RXkrA+RM4ul3horyVhAqccrg1eXmAv24w85BWVeqRGzizwZAQLDLtdS3LCiDmHM+AJtyWneliIZFxJPQrS5g5oXxcS4qRdtLyTgNpJHoeDa+MkIwJ25lFlLwJcq03Sr59Fq9UElAV5/RKmvMCZTzmWSEUDUOJNff4mxcDED7yEnBJ+A24gw6aljaNnRg5w79CYjwB2HeUMB/fwHRf1FZRzFT/0n0Gq63EcvltAmrhL30NGb8Hv3yjeCl41gll2dvP84iT19ZeD19GQs4F04SnuwYydSi68hP3eQ08/7kEzTbUzkDvfwHXgKWH0LmKaB4c5HkU59TND6uF1eFMa4YQ8gd+TtSuA14epaQAh+BE9x3W5GqFu48fX1SXkpzn2K2ak8xwM1bxPFESBNcW/GQHv5EbihXnjRFVwa0wJVUBhWD8GQE/xYuTF4MkfuPDZTcdwxGEMAXuAKTiFdX661Ig/amSA/pRItJh+h+Ao7+d7gWL/magwBuJbgVv7qx6dUsHZpqFTxoa+ua/U3cXU4A6nWVgDhsWChGHDpFIjZNRDPhVykNTdmGCK7cbcOmdWXtRJglue/LrR0vB8W58NEc7vQfHqBkWIijKjGcd4rqXa0WN21gtc43B/OS/++DOUgi0wy0W4FYglT5HGmzvgT25oHQ++VYjFen7yugao18C+NSGi9Bc14JwAAAABJRU5ErkJggg==';

src/routes/llms-full[.]txt.ts

@@ -0,0 +1,30 @@
+import { createFileRoute } from '@tanstack/react-router';
+import { source } from '@/lib/source';
+
+export const Route = createFileRoute('/llms-full.txt')({
+  server: {
+    handlers: {
+      GET: async () => {
+        const pages = source.getPages().sort((a, b) => a.url.localeCompare(b.url, 'en'));
+
+        const rendered = await Promise.all(
+          pages.map(async (page) => {
+            const title = (page.data.title as string | undefined) ?? page.url;
+            const desc = page.data.description as string | undefined;
+            const processed = await page.data.getText('processed').catch(() => '');
+            if (!processed.trim()) return null;
+            const header = desc
+              ? `# ${title} (${page.url})\n\n> ${desc}`
+              : `# ${title} (${page.url})`;
+            return `${header}\n\n${processed.trim()}`;
+          }),
+        );
+
+        const body = rendered.filter((x): x is string => x !== null).join('\n\n---\n\n');
+        return new Response(body, {
+          headers: { 'content-type': 'text/plain; charset=utf-8' },
+        });
+      },
+    },
+  },
+});