Skip to content

Commit 9b949b0

Browse files
committed
Generate LLMS outputs and add HTML injector
Enable generation of site-wide llms.txt/llms-full.txt and per-page markdown (generateMarkdownFiles) so LLM tooling can index docs. Add a new llms-html-injector plugin to normalize generated .md paths, rewrite llms* URLs, and inject <link rel="alternate" type="text/markdown"> tags. Add a verification script (scripts/verify-llms-output.js), ignore its output in .gitignore (/.llms-verify), and remove the hand-maintained static/llms.txt in favor of generated rootContent/fullRootContent. Also update config include/exclude patterns and add related docs/service index updates; package dependencies were bumped (Docusaurus packages) and a new plugin implementation was added at src/plugins/llms-html-injector/index.js.
1 parent 9a85ba1 commit 9b949b0

8 files changed

Lines changed: 766 additions & 445 deletions

File tree

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,9 @@
44
# Production
55
/build
66

7+
# Dev sanity-check output from scripts/verify-llms-output.js
8+
/.llms-verify
9+
710
# Generated files
811
.docusaurus
912
.cache-loader

docusaurus.config.js

Lines changed: 45 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -328,16 +328,47 @@ const config = {
328328
{
329329
// Set docsDir to site root to collect from all directories
330330
docsDir: '.',
331-
// Disable default files since we're generating section-specific files
332-
generateLLMsTxt: false,
333-
generateLLMsFullTxt: false,
331+
// Generate a comprehensive root llms.txt + llms-full.txt covering every page
332+
// (Agent Score: llms-txt-coverage check). The per-section files below remain
333+
// useful as scoped indexes for AI tools.
334+
generateLLMsTxt: true,
335+
generateLLMsFullTxt: true,
336+
// Emit an individual .md file next to every doc page so .md URLs return 200
337+
// (Agent Score: markdown-url-support and llms-txt-directive-md checks). The
338+
// companion llms-html-injector plugin (registered below) normalizes these
339+
// files to URL-aligned paths and injects <link rel="alternate"> tags.
340+
generateMarkdownFiles: true,
341+
// Custom framing for the root index, preserving the prose previously
342+
// hand-maintained in static/llms.txt.
343+
rootContent: `> MetaMask is the leading self-custodial cryptocurrency wallet and Web3 gateway, enabling developers to build dapps that connect to the MetaMask browser extension and mobile app across EVM and Solana ecosystems.
344+
345+
This file follows the [llmstxt.org](https://llmstxt.org/) specification. Section-scoped indexes are also available:
346+
347+
- [MetaMask Connect](https://docs.metamask.io/llms-metamask-connect.txt) ([full](https://docs.metamask.io/llms-metamask-connect-full.txt))
348+
- [Embedded Wallets](https://docs.metamask.io/llms-embedded-wallets.txt) ([full](https://docs.metamask.io/llms-embedded-wallets-full.txt))
349+
- [Smart Accounts Kit](https://docs.metamask.io/llms-smart-accounts-kit.txt) ([full](https://docs.metamask.io/llms-smart-accounts-kit-full.txt))
350+
- [Services](https://docs.metamask.io/llms-services.txt) ([full](https://docs.metamask.io/llms-services-full.txt))
351+
- [Developer dashboard](https://docs.metamask.io/llms-dashboard.txt) ([full](https://docs.metamask.io/llms-dashboard-full.txt))
352+
- [Snaps](https://docs.metamask.io/llms-snaps.txt) ([full](https://docs.metamask.io/llms-snaps-full.txt))
353+
- [Tutorials](https://docs.metamask.io/llms-tutorials.txt) ([full](https://docs.metamask.io/llms-tutorials-full.txt))
354+
`,
355+
fullRootContent: `> MetaMask is the leading self-custodial cryptocurrency wallet and Web3 gateway, enabling developers to build dapps that connect to the MetaMask browser extension and mobile app across EVM and Solana ecosystems.
356+
357+
This file contains the complete MetaMask developer documentation in a single markdown document, following the [llmstxt.org](https://llmstxt.org/) specification.
358+
`,
334359
// Ignore common non-doc directories
335360
// Note: src/pages/** is not ignored so tutorials can be collected
336361
// Each customLLMFiles entry filters by includePatterns, so only matching files are included
337362
ignoreFiles: [
338363
'node_modules/**',
339364
'build/**',
340365
'.docusaurus/**',
366+
'.cursor/**',
367+
'.github/**',
368+
'.husky/**',
369+
'.vscode/**',
370+
'.integrationBuilderCache/**',
371+
'scripts/**',
341372
'static/**',
342373
'src/components/**',
343374
'src/theme/**',
@@ -349,12 +380,18 @@ const config = {
349380
'src/specs/**',
350381
'src/client/**',
351382
'src/scss/**',
383+
// Quickstart "builder" markdown is content fragments embedded by the
384+
// builder UI, not standalone routes; include only the entry pages.
385+
'src/pages/quickstart/builder/**',
386+
'src/pages/quickstart/commonSteps/**',
352387
'i18n/**',
353388
'*.config.js',
354389
'*.json',
355390
'*.lock',
356391
'README.md',
357392
'CONTRIBUTING.md',
393+
'AGENTS.md',
394+
'LICENSE*',
358395
'gator_versioned_docs/**',
359396
],
360397
excludeImports: true,
@@ -481,6 +518,11 @@ const config = {
481518
],
482519
},
483520
],
521+
// Runs in postBuild after docusaurus-plugin-llms generates files. Normalizes
522+
// .md files to URL-aligned paths, rewrites llms*.txt URLs to match, and
523+
// injects <link rel="alternate" type="text/markdown"> into every doc page's
524+
// <head> (Agent Score: llms-txt-directive-html / llms-txt-directive-md).
525+
'./src/plugins/llms-html-injector',
484526
],
485527
clientModules: [require.resolve('./src/client/scroll-fix.js')],
486528
themeConfig:

0 commit comments

Comments
 (0)