Markdown export for every page + AI-bot-aware routing#10
Merged
Conversation
Every page now has a `.md` companion (PostHog-style: append `.md` to any URL). Generated at build time via Astro endpoints reading the same content collections as the HTML pages — single source of truth. What changed: - Per-page `.md` endpoints under `src/pages/` for home, about, pricing, team, privacy, terms, blog index + posts, integrations index + details, dryfit tool + scenarios. - `/llms.txt` curated index and `/llms-full.txt` concatenated bundle, both auto-generated. Replaces the previously hand-maintained `public/llms.txt`. - `<link rel="alternate" type="text/markdown">` in every indexed page so AI agents can discover the markdown version. - Vercel `rewrites` in `vercel.json` that route AI-crawler User-Agents (GPTBot, ClaudeBot, PerplexityBot, etc.) to the `.md` version of the page they requested. Humans and search engines hit the HTML unchanged. Replaces the need for a third-party prerender service like ostr.io — the site is fully static, so prerendering adds latency without value, while raw markdown is what AI bots actually want. Shared helpers in `src/lib/markdown-export.ts` render integration, agent, competitor, and scenario JSON entries into clean Markdown with consistent headers (title, description, canonical link). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.mdto any URL)./llms.txt(curated index, auto-generated) and/llms-full.txt(concatenated bundle)..mdversion of any indexed page via Vercelrewrites— humans and search engines hit the HTML unchanged.Why this instead of ostr.io
The site is fully static (Astro SSG): every page is pre-rendered HTML at build time. Routing bots through ostr.io would add a network hop with no prerendering value to extract. AI bots in particular benefit far more from clean Markdown than from re-rendered HTML, so we serve them the
.mddirectly. Standard search-engine crawlers (Googlebot, Bingbot) keep hitting the static HTML, which is already optimal for them.What's added
/index.mdsrc/data/pages/home.md+integrationscollection/about.md,/privacy.md,/terms.mdsrc/data/pages/*.mdbody/pricing.mdsrc/data/pages/pricing.md+ tiers/addons/FAQ JSON/team.mdteamcollection/blog/index.md,/blog/<slug>.mdblogcollection (raw md body)/integrations/index.md,/integrations/<slug>.mdintegrationsJSON, rendered/tools/dryfit.md+ scenariosdryfit-scenariosJSON, rendered/llms.txt/llms-full.txtPlus:
<link rel="alternate" type="text/markdown">on every indexed HTML page for explicit discoverability, and arobots.txtreference to/llms.txt.Files
src/lib/markdown-export.ts— shared rendering helpers (integration / scenario / competitor / agent JSON → Markdown).src/pages/**/*.md.ts,src/pages/llms.txt.ts,src/pages/llms-full.txt.ts— Astro static endpoints.src/components/seo/Head.astro— adds the markdown alternate link (skipped on noindex pages).vercel.json— addsrewritesblock with UA-conditioned routes.public/llms.txt— deleted (now generated).public/robots.txt— adds reference to/llms.txtand/llms-full.txt.Test plan
/about.md,/blog/we-rebuilt-beton-automated-signal.md,/integrations/posthog.md,/llms.txt,/llms-full.txtand confirmContent-Type: text/markdown(ortext/plainfor.txt) and clean output.User-Agent: GPTBot/1.0against/,/about/,/integrations/posthog/and confirm response body matches the corresponding.mdfile (rewrite working).<link rel="alternate" type="text/markdown" href="...">is present (and absent on/seqd/and/404).🤖 Generated with Claude Code