PoC(WIP): add MDX AST Bridge Implementation for High-Performance Compilation

[jp-knj]

Related PR: #14080, I'm planning to close this later,

Summary

This PR introduces an experimental Rust-based MDX compiler for Astro and maintain compatibility with existing remark and rehype plugins.

Key Features

Performance

• 2-3x faster MDX compilation compared to the JavaScript implementation
• Rust-based parsing and code generation for maximum speed
• Seamless fallback to JS compiler when Rust binary is not available

Plugin Compatibility

• compatible with existing remark and rehype plugins
• Uses an AST bridge pattern: Parse (Rust) → Transform (JS plugins) → Generate (Rust)
• No changes required to existing MDX configurations or plugins

User Experience

• Opt-in via experimental.mdxCompiler: 'rs' configuration

Implementation Details

Architecture

The implementation uses an "AST bridge" pattern that combines the best of both worlds:

This is achieved through the AST Bridge architecture where:

1. Rust handles the heavy parsing (MDX → AST)
2. JavaScript runs the plugins (AST transformations)
3. Rust handles the code generation (AST → JavaScript)

The performance gains are consistent and scalable, making it an excellent choice for projects that need both speed and plugin flexibility.

  packages/
    integrations/
      mdx/
        src/
          processors/
            js.ts         # JavaScript MDX processor
            rust.ts       # Rust MDX processor interface
          rust-parser/    # Rust MDX parser implementation
            Cargo.toml
            src/
              lib.rs
            package.json
            index.js      # Generated by napi-rs

Configuration

  // astro.config.mjs
  export default defineConfig({
    integrations: [mdx()],
    experimental: {
      mdxCompiler: 'rs' // 'js' (default) | 'rs'
    }
  });

Performance Metrics

The Rust-based MDX compiler with plugins delivers:

• Small files (1KB): 3.68x faster
• Medium files (10KB): 2.56x faster
• Large files (100KB): 2.88x faster

Average performance improvement: ~2.5-3.7x faster than the JavaScript compiler while maintaining full compatibility with all remark and rehype plugins.

Technical Considerations

Pros

• Significant performance improvement for content-heavy sites
• No breaking changes or plugin incompatibilities
• Clean abstraction allows future optimizations

Cons

• Requires building native binaries
• Adds complexity to the build process
• Initial setup requires Rust toolchain for contributors

[changeset-bot]

⚠️ No Changeset found

Latest commit: 47d0283

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[codspeed-hq]

CodSpeed Performance Report

Merging #14181 will not alter performance

_{Comparing jp-knj:poc/rust-based-mdx (9b30b53) with main (b36e72f)¹}

Summary

✅ 6 untouched benchmarks

Footnotes

1. No successful run was found on main (9fe883e) during the generation of this report, so b36e72f was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[Eveeifyeve]

Why not ignore this file and put it in a dist?

[jp-knj]

You're right! I'll remove this!

[Eveeifyeve]

Again, https://github.com/withastro/astro/pull/14181/files#r2270673040

[florian-lefebvre]

Hey, thanks a lot for the research! I don't think this should live in Astro, but rather in the upstream package. I'm fine with leaving the PR open if that allows you to do more testing or whatever, otherwise feel free to close it

[Eveeifyeve]

Hey, thanks a lot for the research! I don't think this should live in Astro, but rather in the upstream package. I'm fine with leaving the PR open if that allows you to do more testing or whatever, otherwise feel free to close it

I agree with this, please make the mdx-rs-parser package seperate from the astro repo.

[jp-knj]

Sorry, I'll close.
just creating the package right now

https://github.com/jp-knj/jetmd

[florian-lefebvre]

No need to apologize, I just wanted to set the expectation there. Thanks for the work on this, I think that can be very powerful!