feat(build): native LLM-friendly output (llms.txt) - #5313
Conversation
Adds an experimental `llms` config option that emits, at the end of the build, a raw markdown version of each page (includes expanded, rewrites applied), an llms.txt index following the sidebar order, and an llms-full.txt bundle. Root locale only; dynamic routes are skipped. See https://llmstxt.org/ and #4590
Replaces vitepress-plugin-llms with the built-in `llms: true` option. The generator now detects the landing page through rewrites (en/index.md -> index.md), resolves the sidebar and site metadata through locale/additional config layers via resolveSiteDataByRoute, and renders leaf links before nested sections so trailing top-level links are not misattributed to the previous section heading.
- `llms.ignoreFiles` excludes pages from all LLM output via picomatch globs, matched against both source and rewritten output paths. - `<llm-only>` content appears only in the generated markdown; it is stripped from the rendered HTML. `<llm-exclude>` is the inverse. Both are processed only when the llms option is enabled.
Allows toggling LLM output generation while keeping the rest of the
options, e.g. `llms: { enabled: !!process.env.CI, ... }`. Also gates
the llm-only/llm-exclude tag processing in the markdown pipeline.
hmm, actually the difference is not that big: in vitepress docs (big docs) it's a ~10% improvement. In the screenshot the second run is with llms: true while the first is only with the plugin. This means I had the wrong perception of the speed impact of the plugin (I really thought it was impacting speed much more). I still think |
The approach is wrong. If you want perf gains you need to collect/emit somewhere in between the markdown to vue pipeline. This is currently re-rendering stuff again inside generateLlmsTxt. Also, check the difference with the preview version on pinia/router docs once locally. |
|
/publish |
commit: |
|
In router, locally, the improvement is too small ~200ms. Inside netlify still not enough difference:
|
generateLlmsTxt no longer re-reads pages from disk, re-parses frontmatter, and re-runs include processing with a second markdown renderer: the markdown to vue transform now collects the include-expanded source (before llm tags are stripped for the HTML build) and generateLlmsTxt assembles the output from it.
|
Well, my try was not successful 😆 , the perf is the same but it's more correct (it includes expansions from the build pipeline) |
Yes |
|
I thought we can just move all my plugin code here for now so it basically works, but do as you wish |
If it is possible to port all this to vitepress, I think I will still leave my plugin for maintenance, I think to maintain compatibility with older versions of vitepress |
It's a good chance to reduce the API surface and take a completely different approach, so copying wouldn't be that interesting compared to just having the plugin as an external package 😄 |
|
Will there be any updates on this? |
|
I don't know. Maybe not yet. I don't think this is needed for vitepress 2.0 and I think it's important to keep the initial api surface small (like here) instead of large (like in vitepress-plugin-llms). But that's all @brc-dd 's call |
4 participants
I use https://github.com/okineadev/vitepress-plugin-llms in most of my projects but it takes too much time to generate the files and I think this is because of how it works and probably to enable further, more powerful integrations through remark.
This PR aims to cover what I use in Vue Router, Pinia, Pinia Colada, etc while staying negligible (100ms in this repo docs to the total) in build time and not adding any new dependencies.
Closes #4590
What's included
llms: boolean | LlmsOptionsconfig option (enabled,hostname,title,description,ignoreFiles)index.mdis ignored automaticallyllms.txtindex following the sidebar order,llms-full.txtbundle, and a per-page.mdnext to each.html(includes expanded, rewrites applied)<llm-only>/<llm-exclude>tags to target content at LLMs or humans onlyvitepress-plugin-llmsdependency)Limitations (intentionally not covered)
[param].md) are skipped<<<code snippet imports are not expanded in the markdown output (the directive is left as-is) because the replacement is not exposed where neededllms.txtTOC only understands the default theme's sidebar config<llm-only>/<llm-exclude>are processed with string-level regexes, so literal occurrences inside code fences are also affected@okineadev, are you still interested in maintaining something like this?
PS: @brc-dd this code is fully generated because I needed to see if it was possible to cover what I needed personally. I haven't (at least yet) reviewed the generated code in depth, because I don't know if you want to go forward with this feature or not.