marked v13's token renderer

[emmercm]

Unfortunately marked changed the function signatures on the renderer object.

I have some code to alter renderer behavior:

import {marked} from 'marked';
const markdownRenderer = new marked.Renderer();
markdownRenderer.heading = (text, level, raw) => {
    // ... 
};
markdownRenderer.table = (header, body) => {
    // ...
};
markdownRenderer.code = (_code, infostring, escaped) => {
    // ...
};

and then I provide that renderer like this:

import markdown from '@metalsmith/markdown';
Metalsmith(path.resolve())
    // ...
    .use(markdown({
        renderer: markdownRenderer
    }))
    // ...

markedjs/marked#3291 made it so renderers created with marked v13+ code are incompatible with <v13 versions of marked, and this plugin's version is quite old.

---

Maybe a warning about compatible versions of marked would suffice? I suspect this plugin's version of marked is being held back intentionally based on #70.

An eventual major upgrade of this plugin might be nice, and I'm sure you had other items for its roadmap.

[webketje]

I attempted to update this plugin a year ago and got into contact with the newer maintainer of marked to address issues with the release process and the new extensions ecosystem: https://github.com/search?q=org%3Amarkedjs+type%3Aissue+commenter%3Awebketje&type=issues.

It became clear to me that we are totally at odds about how an open-source project should be managed, see a.o. markedjs/marked#3186 (comment). Metalsmith as a project with regard for backward-compatibility and stability cannot rely on a core plugin that breaks as frequently as bot-automated marked + extension releases. I attempted an upgrade but there were just too many incompatibilities and open questions: https://github.com/metalsmith/markdown/commits/release/2.x. I did a deeper dive in some of the marked extensions and found that for ex the smartypants dependency used has code of dubious quality.

For these reasons I concluded it would be better to switch to an alternative dependency, and stuck with https://github.com/remarkjs/remark, for which I also started work at https://github.com/metalsmith/markdown/commits/poc/remark. Remark can do much more than just markdown, including MDX. I discontinued work on this after 2 hard drive crashes made me lose local progress and I had to focus on home renovations.

[emmercm]

@webketje this was a very thoughtful answer, thank you. I agree with your viewpoint on breaking changes.

Maybe it could be worth a quick README note about this plugin only being compatible with <v13. I'm totally fine with keeping my versions held back.

[webketje]

@emmercm I'm about to tackle this, and was on the fence as to whether to go for marked vs remark in the end.
Marked has popularity, continuity and familiarity going for it, and a history of frequent breaking changes against it.
remark has feature-richness thanks to its plugin-based API, but also brings more complexity.

So I decided to run a local benchmark rendering a representative medium-length markdown file and marked consistently came out 10x faster:

marked x 798 ops/sec ±2.37% (92 runs sampled)
remark x 74.05 ops/sec ±3.11% (66 runs sampled)
Fastest is marked

This difference is unacceptable to me, so I'll stay with marked for perf reasons

[emmercm]

Is that benchmark considering a full-page render as one operation? If so, 75 pages rendered in one second would be plenty fast for most people I would think. My personal site takes more than 2min to build right now, it takes around 6.5sec for my 71 blog pages to render with:

• @metalsmith/default-values
• metalsmith-hbt-md
• metalsmith-mermaid (mine)
• metalsmith-vega (mine)
• @metalsmith/markdown
• metalsmith-excerpts
• @metalsmith/default-values
• metalsmith-feed
• metalsmith-reading-time (mine)

most of that time is probably Mermaid and Vega because they have to use Puppeteer. I'm not sure I would notice a 10x slowdown compared to other plugins that will always take longer, but I understand that's just one voice and you have more to consider with the project.

[webketje]

It may be noticeable at scale... It was just a quick test on a single document, no plugins for either, and only rendering an in-memory string. I had a look at the code for some remark/rehype extensions, they just operate on a more abstract level with more intermediate steps that I am not convinced is optimal for the most common use case.

I think I'm gonna go with marked as default, but provide an example of using remark in the README.md, for example to use ::mdc{#markdown.components}

[wernerglinka]

Hey @webketje + @emmercm

"I've been noodling about a Markdown plugin and want to share thoughts on a unified/remark based approach.

It looks like the unified/remark ecosystem has become a JavaScript community standard with plugins for table of contents, syntax highlighting, GitHub Flavored Markdown, math equations, diagrams, and emoji support. All follow consistent patterns, making extensions straightforward.

I've built a first iteration plugin based on @metalsmith/markdown. Have a look.

Based on initial tests, comparing unified/remark vs. marked shows minimal differences - really just HTML formatting details that don't impact browser rendering or SEO. I don't know about test suites doing exact string matching or custom processors expecting specific HTML formatting.

What do you think about this approach? Have you explored similar solutions? I am still in the middle of testing this and would appreciate any feedback.

[webketje]

Hi @wernerglinka, check out my previous comment in this thread: #73 (comment). The branch poc/remark contains a WIP on remark. Marked & remark have feature parity for basic markdown, extensions and output, but indeed not for more custom use cases.

In the meantime I continued work on that POC locally. There are 3 additional complexities with remark:

• the plugin needs to be completely refactored to default to async rendering by default, which makes it more complex
• a selection of "default plugins" to use needs to be decided & included (also true for new marked, but less than unified)
• marked has a renderInline method which is ideal for keys other than contents that remark lacks

It seems like you have solved the first one (async render), and I may include that. But as mentioned remark is 10x slower without plugins and most users may not need the additional functionality. And even if it's not the default it will be compatible. For ex. I could provide it like so:

import markdown from '@metalsmith/markdown'

// default
metalsmith.use(markdown())
// explicit default
const render = markdown.render.marked
metalsmith.use(markdown({ render }))

// remark
const remarkRender = markdown.render.remark
metalsmith.use(markdown({ render: remarkRender }))

[wernerglinka]

I read your exchange with the marked maintainer. Their attitude is not really confidence-inspiring. I have experienced several times how they, with breaking releases, decide to move functionality out so then you have multiple dependencies instead of just one. But for me, the interesting thing is the internal unified architecture. The ability to do more stuff like syntax highlighting and minimizing HTML all in one plugin sounds interesting - much like a Swiss knife for text processing :-) Reminds me of Metalsmith.

By the way, I built two quick test setups to get an idea of the speed difference. A small site with my new plugin added 9 test pages, including the source of John Gruber's Markdown Syntax page. For the second setup, I duplicated the first and just dropped in @metalsmith/markdown, and it worked. Just a quick test, no options. In any case, as you have observed, the speed difference is about 10:1 in favor of 'marked.`

[wernerglinka]

Update: I added an option to use MicroMark directly. MicroMark is the engine inside Remark, so used alone, it is faster. That reduces the speed difference to 6:1... still a lot.

[webketje]

I looked into micromark but found it not flexible enough, haha. Another path I could go is to publish a @metalsmith/markdown-preset-remark which could be loaded like: ms.use(markdown({ ...presetRemark, ...overrides })).

But right now I'm still leaning more towards marked as default