Resolving references

[andreubotella]

So @flying-sheep started working on resolving references by writing a two-pass transform approach: one which would walk the tree to keep track of every target of a reference, and then one which would walk the tree making the corresponding changes. I'm not sure anymore that all references can be resolved that way.

So, the different types of targets and references in ReST are:

• Footnotes and footnote references (including auto-numbered, auto-symbol, and footnotes with an autonumber label).
• Citations and citation references.and which are linked pairwise in order of appearance in the document.
• Substitution definitions and substitution references. Note that the names for substitutions are not case-insensitive like reference names – they're instead case-sensitive by default, falling back to case-insensitive if there's no match. Fixed in Resolving substitutions. #9.
• Hyperlinks
  • Embedded URIs
  • Explicit hyperlink targets and hyperlink references.
    • internal (linking to a part of the document)
    • external (linking to a URL)
    • indirect (forwarding to another hyperlink target).
  • Anonymous hyperlink references and targets, of which there must be the same number,
  • Hyperlink references can also target other hyperlink references which have an embedded URL, Hyperlink references can target inline internal targets, as well as any implicit targets, which include section titles, footnotes which are either manually numbered or have autonumber labels, and citations. [See my next comment for what docutils actually treats as explicit and implicit targets.]

The thing that makes this hard, perhaps impossible, to resolve in those two passes is substitutions, which in docutils get handled earlier than every other reference. Right now our parser only accepts an image directive inside a substitution definition, but in the general case substitution references can be replaced by any inline content (with the replace directive, for example), which can in turn have auto-numbered footnote references, anonymous hyperlink references or hyperlinks with an embedded URL.

I guess for now we could handle everything in those two passes, but in the future we'll have to move the code for substitutions to its own separate two passes that run before the others.

[flying-sheep]

Excellent breakdown, thank you. The kinds of references in rST were the only part that still confused me, but this seems to be exhaustive! I’ve been documenting the weirder parts of rST/doctrees as I went, this overview should also go somewhere people can find it!

Yeah, doing an earlier pair of passes for substitutions seems like an easy way to fix this in the future.

[andreubotella]

I've also been looking into the system messages that are raised related to the references. This wasn't clearly documented in the spec (indeed, most messages aren't documented at all), and I had to dig into docutils' transforms/references.py, and then do some testing of my own.

The message levels are documented in PEP 258.

Section titles:

• If the target is used for another link target, explicit or implicit, the target is removed. If every other link target with the same name is implicit, that's an info message (level 1).

Footnotes and footnote references:

• If there's a different number of autonumbered or autosymbol footnotes than the corresponding references, that's an error (level 3).
• If a footnote reference refers to a number or autonumber label that isn't defined as a footnote, that's an error (level 3).
• If a footnote uses a duplicate number or a duplicate autonumber label, or one which is also used for another explicit link target, the target is removed, and that's a warning (level 2).
• There's no system message if a footnote isn't referenced.

Citations and citation references:

• If a citation reference refers to a label that isn't defined, that's an error (level 3). (In docutils this isn't an error if the label is a footnote or a hyperlink target, rather than a citation. I suspect this is a bug.)
• If a citation uses a duplicate label, or one which is also used for another explicit link target, the target is removed, and that's a warning (level 2).
• There's no system message if a citation isn't referenced.

Explicit hyperlink targets and references:

• If an indirect target refers to a non-existing link or creates a circular reference, that's an error (level 3).
• If a hyperlink reference refers to a label that isn't defined, that's an error (level 3).
• If there's a different number of anonymous hyperlink targets than the corresponding references, that's an error (level 3).
• If a target uses a label which is also used for another explicit link target, the target is removed, and that's a warning (level 2), except when all targets are external targets with the same URI, in which case that's an info message (level 1).
• If a target isn't referenced, that's an info message (level 1).
• If an internal hyperlink is at the end of the file or followed in tree order by a node of type Comment, SubstitutionDefinition, Pending, Footnote or Citation, the target isn't propagated but no system message is produced.

Inline internal targets:

• If a target uses a label which is also used for another explicit link target, the target is removed, and there's a warning (level 2).
• If a target isn't referenced, that's an info message (level 1).

Substitution definitions and references:

• If a substitution block contains a substitution reference that ends up being circular, or if a substitution reference would expand circularly, that's an error (level 3). (I assume the error is raised both for definitions and references because it's possible to define a custom directive that gets replaced differently depending on context.)
• If a substitution definition has the same name as another substitution definition (compared case-sensitively), that's an error (level 3).
• If a substitution reference refers to a name which isn't defined for a substitution, that's an error (level 3).
• There's no system message if a substitution reference is equal (either case-insensitively or -sensitively) to more than one substitution name: the substitution definition that comes last is used, even if that's a case-sensitve duplicate. (This is underspecified in the spec, and might be a bug in docutils.)
• There's no system message if a substitution definition isn't referenced.

Although the ReST spec treats any link targets other than explicit hyperlink targets and inline internal targets as explicit links, docutils is actually treating any link target other than section titles as explicit. The above list describes how docutils actually does things.

[flying-sheep]

OK, some progress from my side!

• Hyperlinks can now utilize named targets that directly refer to an URL
• I implemented the “replace” directive for substitutions
• I implemented HTML rendering for everything

The README can now be perfectly rendered (almost: There’s no space inserted between two consecutive lines of text)