feat: A native pipeline for content#16149
Open
Princesseuh wants to merge 30 commits into
Open
Conversation
🦋 Changeset detectedLatest commit: 1f15ab6 The changes in this PR will be included in the next version bump. This PR includes changesets to release 428 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
github-actions
Bot
added
feat: markdown
Princesseuh
changed the title
Princesseuh
added
pr preview
github-actions
Bot
removed
the
pr preview
commit: |
Princesseuh
added
the
pr preview
github-actions
Bot
removed
the
pr preview
Princesseuh
added
the
pr preview
github-actions
Bot
removed
the
pr preview
Contributor
📊 Dependency Size ChangesWarning This PR adds 5.8 MB of new dependencies, which exceeds the threshold of 100 kB.
Total size change: 5.8 MB |
github-actions
Bot
added
feat: markdown
Merging this PR will not alter performance
Comparing 
Footnotes
|
Princesseuh
added
the
pr preview
github-actions
Bot
removed
the
pr preview
Princesseuh
added
pr preview
github-actions
Bot
removed
the
pr preview
Princesseuh
force-pushed
the
feat/erika-rust-mdx
branch
from
a9ca5d7 to
bcce353
Compare
github-actions
Bot
added
pkg: svelte
Princesseuh
changed the title
Princesseuh
commented
May 21, 2026
| @@ -0,0 +1,86 @@ | |||
| import yaml from 'js-yaml'; | |||
Member
Author
There was a problem hiding this comment.
I didn't want to change this (this is not a new file) in case not doing it this way would ripple through a million files, but some (Sätteri) Markdown processors can return both the frontmatter and the content in one very fast call, so extracting it like that is inefficient.
Princesseuh
commented
May 21, 2026
Comment on lines
+17
to
+18
| const firstPos = 'position' in first ? first.position : undefined; | ||
| const lastPos = 'position' in last ? last.position : undefined; |
Member
Author
There was a problem hiding this comment.
This seems unrelated at first glance, but our Vite plugin for .html imports use rehype, so global augmentation to the hast module by Sätteri and Rehype ripples here.
Princesseuh
commented
May 21, 2026
| markdown: { | ||
| syntaxHighlight: false, | ||
| // Keep straight quotes so assertions can match `import.meta.env` output literally | ||
| processor: satteri({ features: { smartPunctuation: false } }), |
Member
Author
There was a problem hiding this comment.
Sätteri's smartypants implementation is better than the popular remark plugin, this caused the test to fail because it expected quotes to stay straight, see https://satteri.bruits.org/docs/divergences/#smart-punctuation-pairing-across-nodes
Princesseuh
commented
May 21, 2026
Comment on lines
+87
to
+94
| // Coerce element/null to boolean before asserting — assert.equal on a raw | ||
| // linkedom Element vs null pulls util.inspect through the DOM's circular | ||
| // refs and can take down the test process on failure. | ||
| assert.equal( | ||
| selectRemarkExample(document) !== null, | ||
| true, | ||
| 'MDX remark plugins not applied.', | ||
| ); |
Member
Author
There was a problem hiding this comment.
This is the thing that I wasted a day chasing a memory leak for, if the previous test failed, it would instantly take 25gb of RAM and crash everything. This form avoids this problem
Princesseuh
commented
May 21, 2026
| while ((match = regex.exec(html)) !== null) { | ||
| const matchKey = ${rawUrl} + '_' + occurrenceCounter; | ||
| const imageProps = JSON.parse(match[1].replace(/"/g, '"').replace(/'/g, "'")); | ||
| const imageProps = JSON.parse(match[1].replace(/&(?:#x22|quot);/g, '"').replace(/&(?:#x27|apos);/g, "'")); |
Member
Author
There was a problem hiding this comment.
Same thing regarding named vs number entities.
Princesseuh
commented
May 21, 2026
| @@ -0,0 +1,47 @@ | |||
| // TODO: This is a workaround around a missing API in Sätteri. The visitor architecture naturally does not provide | |||
Member
Author
There was a problem hiding this comment.
This will be gone before the actual release, it's not a bug or anything, just haven't decided of an API shape for this usecase yet.
Princesseuh
commented
May 21, 2026
| @@ -0,0 +1,158 @@ | |||
| import type * as hast from 'hast'; | |||
Member
Author
There was a problem hiding this comment.
I moved all the processor-agnostic Markdown stuff into this package, because it was the common point of import between everything (and cannot live in Astro itself, recursive deps and all that). I don't think it's a problem, but we could also have some sort of @astrojs/internal-markdown dedicated to this, but I didn't see the point.
Princesseuh
commented
May 21, 2026
|
|
||
| The top-level `markdown.remarkPlugins`, `markdown.rehypePlugins`, and `markdown.remarkRehype` options are deprecated. They'll continue to work when `@astrojs/markdown-remark` is installed for now, but this will be removed in the next major. | ||
|
|
||
| The top-level `markdown.gfm` and `markdown.smartypants` options are also deprecated. Move them onto your processor instead — `satteri({ features: { gfm: false, smartPunctuation: false } })`, or `unified({ gfm: false, smartypants: false })`: |
Member
Author
There was a problem hiding this comment.
In theory, those two options could be fully removed and we could ask users to manually install the Remark plugins, but I figured for a better migration story right now we can just support them in the processor itself.
It's not relevant for Sätteri because it's supported natively there.
github-actions
Bot
added
pkg: vue
ocavue
reviewed
May 21, 2026
ocavue
reviewed
May 21, 2026
matthewp
reviewed
May 22, 2026
|
|
||
| export default defineConfig({ | ||
| markdown: { processor: satteri() }, | ||
| integrations: [mdx({ processor: unified({ remarkPlugins: [/* ... */] }) })], |
Contributor
There was a problem hiding this comment.
The only thing a little weird here, if you wanted to use unified for both, would that mean you'd do:
import { defineConfig, satteri } from 'astro/config';
import mdx from '@astrojs/mdx';
import { unified } from '@astrojs/markdown-remark';
export default defineConfig({
markdown: { processor: unified({ remarkPlugins: [/* ... */] }) },
integrations: [mdx({ processor: unified({ remarkPlugins: [/* ... */] }) })],
});Or would you keep the processor in a variable and reference it in both places? Do they need to share state?
Member
Author
There was a problem hiding this comment.
No, it's the same as the current options, MDX inherits the options from markdown by default: https://docs.astro.build/en/guides/integrations-guide/mdx/#extendmarkdownconfig
It's a weird behavior of MDX in my opinion... but in line with what we currently have, at least. They don't need to share state, the things that could be shared are always module-level (e.g. Shiki)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Implements withastro/roadmap#1364
Testing
Added tests, updated tests that depends on remark behavior to use the unified pipeline
Docs
withastro/docs#13919