Support for custom content format renderers and transformations

[rgov]

There are some kinds of transformations that I'd like to be able to perform on my content that are not possible or easy with shortcodes.

For instance, it is not currently possible to wrap each rendered section of a Markdown document from its # heading to its last paragraph in a <section> tag, or to change how footnotes are rendered, or to embed a Graphviz diagram in my content file and have it rendered to an embedded SVG by invoking dot, etc. These kinds of changes would require patching Hugo.

Now that Hugo has the ability to execute external tools like Asciidoctor, why not make it more flexible so that users can provide their own content format renderers?

1. Hugo publishes a specification on how a renderer is invoked and how it should return rendered content (e.g., the body of the rendered content, and a list of anchors for the table of contents).

2. The user configures how the renderer is invoked, such as: the tool's path; the file extension for the content; whether to perform further shortcode expansion on the output.

(Rather than allow a complex configuration of command line arguments for an existing tool like Asciidoctor, the user would provide an adapter script.)

This extends nicely to supporting content pipelines (unrelated to existing asset pipelines). The default pipeline for .md files might look like this:

1. Parse file content with Goldmark
2. Generate HTML
3. Perform shortcode expansion
4. Output

If I want to customize this pipeline, I could use the same building blocks, and insert an external transformation (or multiple) that modifies the Markdown before it gets rendered by Goldmark, or before it renders to HTML, etc.

[rgov]

Pipelines were discussed in #3207 but the discussion focused on assets (images, etc.) not content.

A similar proposal is to customize how external helpers are invoked to support different output formats (#6089). The proposal in this issue would be able to achieve this as well, if the output steps of the pipeline can be overridden (for example, to add an HTML to LaTeX step).

Plugins have been discussed but the issue brought up is security. Since this is configured entirely by the user, and not by third-party templates or content, it doesn't have the same drawbacks as #796. (Though, if someone wished, they would be able to implement an exec shortcode for themselves using this proposal, by adding a custom transformation.)

Also, by allowing for external transformation tools, they do not need to all be written in Go. This makes it more attractive than true plug-ins for people who are familiar with languages such as JavaScript or Python.

[rigtorp]

This is related to #7765 Add support for generating figures and diagrams.