QVAC-18431 Docs: markdown negotiation - support for AI agents (#2375)

BrunoCampana · web-flow · commit 92ee4f0af91e · 2026-06-03T07:12:20.000-03:00
* doc: fix broken redirect rules

* doc: add fixed redirects for versioned docs

* doc: fix versioned docs - readd hardcoded rules as sevalla doesnt handle splats correctly

* doc: test to use placeholders rather than hardcoded using sevalla syntax

* doc: fix versioned docs - readd hardcoded rules as sevalla doesnt handle splats correctly

* doc: test redirect for markdown negotiation

* doc: test markdown negotiation, with only hardcoded paths

* doc: test markdown negotiation with splats

* doc: markdown negotiation tests - placeholders

* doc: final test for markdown negotiation: validating status code

* doc: standardize trailing slash in the end in all URLs

* doc: testing deduplication of URLs without trailing slash

* doc: redirects - test - add protection for URLs with dot

* doc: redirects - test - markdown negotiation - with correct syntax

* doc: redirects - test - markdown negotiation - with correct syntax

* doc: redirects - test - markdown negotiation - with correct syntax

* doc: redirects - test - fix versioned paths

* doc: fix markdown missing .md files being generated - versioned API pages

* doc: add comments in redirects to remember why it was required to add the 404 block atthe bottom
diff --git a/docs/website/public/_redirects b/docs/website/public/_redirects
@@ -1,127 +1,158 @@
-# Markdown content negotiation — serve the .md sibling of a page when the
-# client asks for it via `Accept: text/markdown`. Home maps to `/index.md`;
-# every other page `/foo/bar/` maps to `/foo/bar.md`. We need both the
-# trailing-slash and no-slash variants so the `:splat` placeholder doesn't
-# produce `/foo/.md`.
-# /                /index.md      301   Header:Accept=text/markdown
-# /*/              /:splat.md     301   Header:Accept=text/markdown
-# /*               /:splat.md     301   Header:Accept=text/markdown
+# Markdown content negotiation — serve the `.md` sibling of a page when
+# the client requests it via `Accept: text/markdown`. Home maps to
+# `/index.md`; every other page `/a/b/.../z/` maps to `/a/b/.../z.md`.
+#
+# Three properties of these rules and why they are required:
+#
+#  1. `!` (force) — the source path has an underlying static file (the
+#     HTML page) that would otherwise be served; the force flag is what
+#     lets the rule fire at all.
+#
+#  2. `303` (See Other) — the rule MUST be a 3xx redirect, not a 2xx
+#     rewrite. With a 2xx rewrite the response body for the `.md` content
+#     gets cached by Cloudflare under the page's URL key (Cloudflare does
+#     not honour `Vary: Accept`), poisoning the cache for subsequent
+#     HTML clients. A redirect is fetched anew by the client, so the
+#     Markdown response is cached under the `.md` URL key and the HTML
+#     response stays cached under the page URL key. Among 3xx codes,
+#     `303` is the semantically correct match for content negotiation
+#     ("the representation you asked for lives at another URI").
+#
+#  3. One rule per nesting depth, no splat. `_redirects` placeholders
+#     (`:name`) match a single path segment — verified empirically
+#     against Sevalla (a single `/:slug/` rule served HTML, not Markdown,
+#     for `/ai-capabilities/text-generation/`). Covering the site
+#     therefore requires one rule per depth. The splat form
+#     (`/* /:splat.md`) would be greedy but empirically crashes the
+#     Sevalla edge worker (HTTP 1101) when combined with `Header:Accept`
+#     — bug in their parser. Current site max depth is 3 segments; one
+#     extra level is included as headroom.
+#
+# Loop safety: the source pattern always ends in `/`, the rewrite target
+# always ends in `.md` (no slash). Those two URL spaces are disjoint, so
+# the client's follow-up `GET /foo.md` does not re-match the rule. This
+# only holds because all incoming page requests carry a trailing slash
+# (Sevalla's Pretty URLs normalizes bare paths first) AND because Pretty
+# URLs skips paths whose final segment contains a dot — so `/foo.md` is
+# never normalized into `/foo.md/`, which would otherwise loop.
+/                       /index.md                  303!   Header:Accept=text/markdown
+/:a/                    /:a.md                     303!   Header:Accept=text/markdown
+/:a/:b/                 /:a/:b.md                  303!   Header:Accept=text/markdown
+/:a/:b/:c/              /:a/:b/:c.md               303!   Header:Accept=text/markdown
+/:a/:b/:c/:d/           /:a/:b/:c/:d.md            303!   Header:Accept=text/markdown
 
 # Sevalla CDN does not resolve directory-to-index.html for path segments
 # containing dots (e.g. `v0.7.x`). The `:version` placeholder matches a
-# single path segment between slashes — so the trailing-slash variant
-# captures the version without the trailing slash, avoiding the double-
-# slash issue that the splat (`*`) has.
+# single path segment between slashes; the `200` rule rewrites each
+# versioned page's trailing-slash form to its `index.html`.
+#
+# Sevalla's Pretty URLs feature normally 301-redirects bare paths to the
+# trailing-slash form, but empirically it SKIPS paths whose final segment
+# contains a dot (treats them as file requests). So we cannot rely on it
+# for `vX.Y.Z` segments — we provide an explicit bare→with-slash 301
+# below.
+#
+# ORDER MATTERS (first-match-wins). The `200` rewrites for the
+# with-slash source MUST come before the `301` redirects for the bare
+# source. Sevalla's placeholder matcher is lenient about trailing slash
+# in the source pattern: a source ending in `:version` (no slash) will
+# match both `/foo/X` AND `/foo/X/`. If the `301` rule appeared first,
+# it would match the with-slash request too and 301 to itself — an
+# infinite redirect loop (verified empirically). Putting the `:version/`
+# rule first, with literal trailing slash, makes the more-specific
+# pattern win for with-slash requests, leaving the `301` rule to handle
+# only the bare form.
 #
 # Syntax mirrors the doc example
 # (https://docs.sevalla.com/static-sites/redirects#placeholders):
 #   /store/:category/:item   /products/:category/:item
-# No explicit status code — defaults to 301 redirect.
-
-/reference/api/:version/   /reference/api/:version/index.html   200
-/reference/api/:version    /reference/api/:version/index.html   200
 
+/reference/api/:version/             /reference/api/:version/index.html             200
 /reference/release-notes/:version/   /reference/release-notes/:version/index.html   200
-/reference/release-notes/:version    /reference/release-notes/:version/index.html   200
+/reference/api/:version              /reference/api/:version/                       301
+/reference/release-notes/:version    /reference/release-notes/:version/             301
 
 # ======================================================================
 # PERMANENT REDIRECTS DUE TO CONTENT MOVE
 # ----------------------------------------------------------------------
 # Everything above is pattern-based (wildcards / :placeholders).
 # Everything below is a 1:1 path move kept for backward compatibility
 # with links from qvac.tether.io and older bookmarks (301 → current IA).
+# Sevalla's Pretty URLs 301-normalizes bare-path requests to the
+# trailing-slash form before these rules run, so only the with-slash
+# variants are listed.
 # ======================================================================
 
 # Section rename: /about-qvac/* → /about/* (+ deletions folded into home)
 /about-qvac/welcome/         /                            301
-/about-qvac/welcome          /                            301
 /about-qvac/flagship-apps/   /                            301
-/about-qvac/flagship-apps    /                            301
 /about-qvac/how-it-works/    /about/how-it-works/         301
-/about-qvac/how-it-works     /about/how-it-works/         301
 /about-qvac/public-launch/   /about/public-launch/        301
-/about-qvac/public-launch    /about/public-launch/        301
 /about-qvac/vision/          /about/vision/               301
-/about-qvac/vision           /about/vision/               301
 
 # Section rename: /sdk/getting-started/* → top-level guides
 /sdk/getting-started/                /introduction/    301
-/sdk/getting-started                 /introduction/    301
 /sdk/getting-started/quickstart/     /quickstart/      301
-/sdk/getting-started/quickstart      /quickstart/      301
 /sdk/getting-started/installation/   /installation/    301
-/sdk/getting-started/installation    /installation/    301
 /sdk/getting-started/configuration/  /configuration/   301
-/sdk/getting-started/configuration   /configuration/   301
 
 # Section rename: /sdk/examples/ai-tasks/* → /ai-capabilities/*
 /sdk/examples/ai-tasks/completion/         /ai-capabilities/text-generation/    301
-/sdk/examples/ai-tasks/completion          /ai-capabilities/text-generation/    301
 /sdk/examples/ai-tasks/fine-tuning/        /ai-capabilities/fine-tuning/        301
-/sdk/examples/ai-tasks/fine-tuning         /ai-capabilities/fine-tuning/        301
 /sdk/examples/ai-tasks/image-generation/   /ai-capabilities/image-generation/   301
-/sdk/examples/ai-tasks/image-generation    /ai-capabilities/image-generation/   301
 /sdk/examples/ai-tasks/multimodal/         /ai-capabilities/multimodal/         301
-/sdk/examples/ai-tasks/multimodal          /ai-capabilities/multimodal/         301
 /sdk/examples/ai-tasks/ocr/                /ai-capabilities/ocr/                301
-/sdk/examples/ai-tasks/ocr                 /ai-capabilities/ocr/                301
 /sdk/examples/ai-tasks/rag/                /ai-capabilities/rag/                301
-/sdk/examples/ai-tasks/rag                 /ai-capabilities/rag/                301
 /sdk/examples/ai-tasks/text-embeddings/    /ai-capabilities/text-embeddings/    301
-/sdk/examples/ai-tasks/text-embeddings     /ai-capabilities/text-embeddings/    301
 /sdk/examples/ai-tasks/text-to-speech/     /ai-capabilities/text-to-speech/     301
-/sdk/examples/ai-tasks/text-to-speech      /ai-capabilities/text-to-speech/     301
 /sdk/examples/ai-tasks/transcription/      /ai-capabilities/transcription/      301
-/sdk/examples/ai-tasks/transcription       /ai-capabilities/transcription/      301
 /sdk/examples/ai-tasks/translation/        /ai-capabilities/translation/        301
-/sdk/examples/ai-tasks/translation         /ai-capabilities/translation/        301
 /sdk/examples/ai-tasks/voice-assistant/    /ai-capabilities/voice-assistant/    301
-/sdk/examples/ai-tasks/voice-assistant     /ai-capabilities/voice-assistant/    301
 
 # Section rename: /sdk/examples/p2p/* → /p2p-capabilities/*
 /sdk/examples/p2p/blind-relays/          /p2p-capabilities/blind-relays/          301
-/sdk/examples/p2p/blind-relays           /p2p-capabilities/blind-relays/          301
 /sdk/examples/p2p/delegated-inference/   /p2p-capabilities/delegated-inference/   301
-/sdk/examples/p2p/delegated-inference    /p2p-capabilities/delegated-inference/   301
 
 # Section rename: /sdk/examples/utilities/* → /runtime/*, /models/*, /configuration/plugins/*
-/sdk/examples/utilities/logging/              /runtime/logging/                          301
-/sdk/examples/utilities/logging               /runtime/logging/                          301
-/sdk/examples/utilities/profiler/             /runtime/profiler/                         301
-/sdk/examples/utilities/profiler              /runtime/profiler/                         301
-/sdk/examples/utilities/runtime-lifecycle/    /runtime/lifecycle/                        301
-/sdk/examples/utilities/runtime-lifecycle     /runtime/lifecycle/                        301
-/sdk/examples/utilities/download-lifecycle/   /models/download-lifecycle/                301
-/sdk/examples/utilities/download-lifecycle    /models/download-lifecycle/                301
-/sdk/examples/utilities/sharded-models/       /models/sharded-models/                    301
-/sdk/examples/utilities/sharded-models        /models/sharded-models/                    301
-/sdk/examples/utilities/plugin-system/        /configuration/plugins/                    301
-/sdk/examples/utilities/plugin-system         /configuration/plugins/                    301
+/sdk/examples/utilities/logging/              /runtime/logging/                            301
+/sdk/examples/utilities/profiler/             /runtime/profiler/                           301
+/sdk/examples/utilities/runtime-lifecycle/    /runtime/lifecycle/                          301
+/sdk/examples/utilities/download-lifecycle/   /models/download-lifecycle/                  301
+/sdk/examples/utilities/sharded-models/       /models/sharded-models/                      301
+/sdk/examples/utilities/plugin-system/        /configuration/plugins/                      301
 /sdk/examples/utilities/write-custom-plugin/  /configuration/plugins/write-custom-plugin/  301
-/sdk/examples/utilities/write-custom-plugin   /configuration/plugins/write-custom-plugin/  301
 
 # Section rename: /sdk/api → /reference/api, /sdk/release-notes → /reference/release-notes
 /sdk/api/                       /reference/api/                       301
-/sdk/api                        /reference/api/                       301
 
 # Section rename: /sdk/tutorials/* → /tutorials/*
 /sdk/tutorials/electron/   /tutorials/electron/   301
-/sdk/tutorials/electron    /tutorials/electron/   301
 /sdk/tutorials/expo/       /tutorials/expo/       301
-/sdk/tutorials/expo        /tutorials/expo/       301
 
 # Section rename: /http-server → /cli/http-server
 /http-server/   /cli/http-server/   301
-/http-server    /cli/http-server/   301
 
 # ======================================================================
 # CATCH-ALL 404 — MUST BE THE LAST RULE
 # ----------------------------------------------------------------------
 # Sevalla serves /404.html automatically for unresolved paths, but with
 # HTTP 200, which breaks SEO, link checkers, analytics and HTTP clients.
 # This explicit rule forces a real `404 Not Found` status while still
-# rendering the same /404.html body. Only reached when no static file and
-# no rule above matches.
+# rendering the same /404.html body. Reached in two distinct cases:
+#
+#   1. No static file matches the request AND no earlier rule matches.
+#   2. An earlier `200` rewrite points at a target that doesn't exist
+#      (e.g. `/reference/api/notaversion/` → rewrite to
+#      `/reference/api/notaversion/index.html` which isn't built).
+#      Sevalla's pipeline continues evaluating rules in this case, and
+#      this catch-all gives those orphaned rewrites a clean 404 instead
+#      of falling back to Sevalla's soft-200 default.
+#
+# Known edge case (intentionally not fixed): a direct request to
+# `/404.html` returns 200 because Sevalla's static-file resolution runs
+# before `_redirects` and serves the file as-is, bypassing this rule.
+# `/404.html` is not in the sitemap, not linked from any page, and not
+# referenced in any external surface; practical traffic to it is zero.
 # ======================================================================
 
 /*   /404.html   404
diff --git a/docs/website/scripts/generate-llm-md-files.ts b/docs/website/scripts/generate-llm-md-files.ts
@@ -18,13 +18,20 @@
  * this script handles the layout entirely in user-space.
  *
  * URL → file mapping:
- *   '/'                    → out/index.md
- *   '/quickstart'          → out/quickstart.md
- *   '/reference/api'       → out/reference/api.md
+ *   '/'                            → out/index.md
+ *   '/quickstart'                  → out/quickstart.md
+ *   '/reference/api'               → out/reference/api.md
+ *   '/reference/api/v0.10.x'       → out/reference/api/v0.10.x.md  (archived)
  *
- * Archived per-section versions (e.g. `/reference/api/v0.7.0`) are not in
- * the manifest, so they are not emitted as `.md` either — consistent with
- * `llms.txt`, `llms-full.txt`, `sitemap.xml`, and per-page `noindex`.
+ * Archived per-section versions ARE included in the manifest. The HTML
+ * for those pages renders publicly (with `noindex` + canonical-to-latest
+ * for SEO posture); the per-page `.md` is just its Markdown
+ * representation, so it must exist for the in-page "Copy as Markdown"
+ * action and the `Accept: text/markdown` content-negotiation flow
+ * (configured in `public/_redirects`) to resolve cleanly. The aggregate
+ * catalogs (`llms.txt`, `llms-full.txt`, `sitemap.xml`) keep filtering
+ * archives — see the comment in `llm-md-manifest.json/route.ts` for the
+ * rationale.
  *
  * Usage (invoked from `package.json` after `next build`):
  *   bun run scripts/generate-llm-md-files.ts
diff --git a/docs/website/src/app/llm-md-manifest.json/route.ts b/docs/website/src/app/llm-md-manifest.json/route.ts
@@ -1,6 +1,5 @@
 import { source } from '@/lib/source';
 import { getLLMText } from '@/lib/get-llm-text';
-import { isArchivedPage } from '@/lib/docs-open-graph';
 
 // Resolves the response at build time so the result is written to
 // `out/llm-md-manifest.json` as a static file under `output: 'export'`.
@@ -9,19 +8,50 @@ export const revalidate = false;
 
 /**
  * Internal build-time data dump consumed by
- * `scripts/generate-llm-md-files.ts`. Emits one entry per non-archived page
- * with the processed Markdown body (same format as `llms-full.txt` chunks);
- * the post-build splitter reads it, writes one `out/<slug>.md` per entry,
- * and then deletes the manifest so it never ships to the CDN.
+ * `scripts/generate-llm-md-files.ts`. Emits one entry per page (including
+ * archived versions of indexable sections — see below) with the processed
+ * Markdown body; the post-build splitter reads it, writes one `out/<slug>.md`
+ * per entry, and then deletes the manifest so it never ships to the CDN.
  *
  * This indirection exists because `output: 'export'` does not support
  * `rewrites()` and Next.js does not allow `.md` as part of a dynamic route
  * segment (e.g. `[[...slug]].md/route.ts` is invalid). A JSON dump consumed
  * by a tiny splitter gives us predictable file naming with no `out/...`
  * staging tree to clean up.
+ *
+ * Policy — archived pages ARE included
+ * ------------------------------------
+ * Unlike `sitemap.xml`, `llms.txt`, and `llms-full.txt`, this manifest does
+ * not filter out archived pages (`isArchivedPage`). The earlier policy was
+ * to suppress every "AI-friendly" representation of archived sections that
+ * carry `noindex` + canonical-to-latest (currently `API_SECTION`), but
+ * conflating "not in aggregate catalogs" with "no per-page `.md`" hurts UX
+ * for two distinct callers:
+ *
+ * - The in-page "Copy as Markdown" action issues `fetch(`${pageUrl}.md`)`.
+ *   When the `.md` is missing the button silently 404s, even though the
+ *   HTML page renders fine.
+ * - AI agents performing per-page Markdown content negotiation
+ *   (`Accept: text/markdown`) get redirected by `public/_redirects` to a
+ *   `.md` URL that does not resolve, ending the chain in a broken 404.
+ *
+ * Per-page `.md` is just an alternate representation of an already-public
+ * HTML page (the archive HTML still renders, with `noindex` carrying the
+ * SEO signal). It is conceptually different from the aggregate catalogs
+ * (`llms.txt` / `llms-full.txt`), which are bulk training-corpus artefacts
+ * where excluding archives genuinely reduces cross-version AI ingestion.
+ *
+ * Net effect of including archived pages here:
+ *   - "Copy as Markdown" works on every page that renders HTML.
+ *   - Markdown content negotiation in `_redirects` resolves cleanly because
+ *     the static `.md` file always exists alongside the static HTML.
+ *   - SEO posture is unchanged: archives stay out of `sitemap.xml`, carry
+ *     `noindex`, and point their canonical at the latest series.
+ *   - Bulk AI training corpus posture is unchanged: archives remain
+ *     excluded from `llms.txt` and `llms-full.txt`.
  */
 export async function GET() {
-  const pages = source.getPages().filter((page) => !isArchivedPage(page));
+  const pages = source.getPages();
 
   const entries = await Promise.all(
     pages.map(async (page) => ({
diff --git a/docs/website/src/lib/docs-json-ld.ts b/docs/website/src/lib/docs-json-ld.ts
@@ -78,7 +78,7 @@ function buildBreadcrumbList(slugs: string[]): JsonLdBlock {
       '@type': 'ListItem',
       position: i + 2,
       name: slugs[i],
-      item: `${DOCS_SITE_ORIGIN}${accumulatedPath}`,
+      item: `${DOCS_SITE_ORIGIN}${accumulatedPath}/`,
     });
   }
 
diff --git a/docs/website/src/lib/docs-open-graph.ts b/docs/website/src/lib/docs-open-graph.ts
diff --git a/docs/website/tests/docs-open-graph.test.ts b/docs/website/tests/docs-open-graph.test.ts

Original file line number	Diff line number	Diff line change
`@@ -78,7 +78,7 @@ function buildBreadcrumbList(slugs: string[]): JsonLdBlock {`
`78`	`78`	`'@type': 'ListItem',`
`79`	`79`	`position: i + 2,`
`80`	`80`	`name: slugs[i],`
`81`		- item: `${DOCS_SITE_ORIGIN}${accumulatedPath}`,
	`81`	+ item: `${DOCS_SITE_ORIGIN}${accumulatedPath}/`,
`82`	`82`	`});`
`83`	`83`	`}`
`84`	`84`