Support for code-specific styles via Pygments; multiple code styles in one document

[hmedina]

Use case & problem

I have a document that displays code in two different languages. Both are idiosyncratic and do not conform to the standard tokens managed by Pygments, and so I wrote them lexers, tokens, and stylers (hosted at hmedina/Pygments_Kappa_plugin). As that plugin utilizes the Pygments API to declare its components, it works fine with the pygmentize program and the rest of that stack. However, for Sphinx, the restriction of "one style per document" (see #9105 (comment)) is a problem, as the style for one language does not apply cleanly to the other. I also don't see a reason why it should apply cleanly; different languages express different things, and so in the general sense, their styles can be specific to whatever they're trying to express.

Desired experience

I would like an experience similar to what I had when using LaTeX and Minted, where each code block gets its own language and style specification, rather than having a universal style for all code blocks across the document.

Path forward?

It appears to me an appropriate point to add this functionality to is the literalinclude directive and its siblings (i.e. code-block, sourcecode, code), as they already allow one to specify the language (but not it's style). The way Minted handles scope, it creates a unique string for each code block, and scopes settings to that. That approach could work for Sphinx, in keeping with a single CSS sheet. Before I go diving into the code, I am looking to start a conversation; does this seem like useful to others? should I look at other parts? are there stumbling blocks I should be aware of? are there easier ways to achieve this?

[hmedina]

I've posted some work in my fork at https://github.com/hmedina/sphinx , focusing on the HTML side. The prototype is working.

The affected directives get an option that serves as an override for the default style. Actually, two options, one for light style, another for dark style (whether a theme uses both is theme dependent; Sphinx already tracks the light and dark Pygments separately). These then serve to add elements to the CSS file, using selectors to override defaults.

For a sample document:

[hmedina]

For HTML builds, Sphinx handles dark/light modes in two different CSS files. What's the rationale for this?
Is there a reason to avoid using @media queries, a single CSS pygments file, and letting user-agents / OS setting pick? (see https://developer.mozilla.org/en-US/docs/Web/CSS/color-scheme)

[hmedina]

From seeing the usage in the official python_docs_theme, it seems avoiding the @media queries and setting the view mode via javascript is a more appropriate route, as it allows users to set their preferred view mode as appropriate.

The theme is capable of overriding the appropriate code pods, and respond to the theme switch:

Light vs Dark view

[hmedina]

I've tested the rest of the builders (except AppleHelp and DocTest), and results are good. For most, this has no impact as they did not support highlighting in the first place (e.g. man, TexInfo, GetText). The LaTeX and HTML family builders appear to work correctly.

[jfbu]

(continued from #13611 (comment))

Two styles that define a background color but do not assign a text color to the root Token are manni and gruvbox-light. Neither has invisible text; both appear to behave as expected, with black text.

In PDF, the text is per default black in code-blocks. On light backgrounds this shows well. I had experimented in the past in PDF output with Pygments styles suitable for dark backgrounds. I am really short on time now, but my memory was that the Pygments output LaTeX mark-up wasn't completely equivalent to what is achieved in HTML/CSS, and it could be observed that some tokens were not attributed a light color and were basically not visible on Pygments style background color. Using the root token color attribute as per you suggested and which I implemented via a suitable configuration of the text color in the code-block appears to be a good automatic solution (I think it will also affect line numbers if the lines are numbered which may be good or bad). When time permits I can check all 70 or so styles I have installed at my place. Extensive testing however can be done only if the PR is real candidate for merging upstream.

While I agree the light-gray background looks better, I would prefer to respect the theme's choice of background.

In my opinion, it would look very weird if all especially styled blocks were to obey in PDF this choice of "white" background. This would contrast with the non-especially styled blocks. To re-enforce some kind of uniformity one would have to globally set the VerbatimColor key of 'sphinxsetup' to white.

The root token from Pygments can have a bg:#123 color specified. Both it and Style.background_color appear to alter the same core behavior, but the former overrides the latter. This said, none of the stock Pygments styles use a background color for the root Token, and since its effects appear better set via Style.background_color, I believe we wont have to deal with it; i.e. there's no need to do something with the second capturing group.

Thanks for investigation. This is good news.

[jfbu]

In my opinion, it would look very weird if all especially styled blocks

I meant those specifying "white" as background; probably simply because the author of the style saw no better default. But in PDF output we have a very light gray as default. English is not my native language and it takes at times too much time for me to search for appropriate explanation

[hmedina]

Responding here to @AA-Turner

General note: I apologise for not having time to review, but I am not entirely convinced of the need for this feature. As far as I can tell, it hasn't been requested before, and the use-case here is for both a custom/private language and a custom (non-Pygments) lexer. This introduces a fair amount of complexity whilst we're undertaking a large refactoring project to reduce inter-linking in the internals of Sphinx.

Currently, Sphinx does not support using more than one style for code blocks, even if these are Pygments styles with Pygments lexers and Pygments tokens. Such a restriction appeared to be more of "nobody has built it yet" rather than based on principle. This feature is already present in other toolchains (e.g. LaTeX & Minted, Typst).

Allowing more than one style per document increases the expressive power of this tool. It mirrors how admonitions can render with different styles (e.g. green tips vs. blue rationales vs. yellow warnings). Beyond my own use-case, I could imagine using style overrides to distinguish compiler versions for code that's "the same language" (gcc vs. msvc ? pascal vs. delphi ?), to further highlight the difference between languages that may not be obvious at first glance, or even for syntactically-valid code that's bad practice.

As I link in the PR, a brief comment can be found at #9105 (comment)
Given there are alternative toolchains that support multiple-styles per document, it is unclear how many projects may have avoided Sphinx due to this limitation.

I hope the above is convincing in terms of the value added by this feature. In terms of the implementation, I am aware every line added adds a maintenance cost; if there are alternative implementations that would be easier to maintain, I'm more than open to suggestions.