Endnotes support in DOCX, ODT and markdown (via endnotes extension)

[massifrg]

PR #11501 provides endnotes support for docx, both in the Writer and in the Reader.
It is activated by the endnotes extension, which is disabled by default.

I'm starting this thread to collect some feedback.

Since endnotes are not in pandoc types, I'm emulating them with this convention:

A Note is to be considered an endnote when it's embedded in a Span that has only one class: "endnote".
(only when the endnotes extension is active!)

Here's an example in markdown:

A paragraph with an endnote[[^E1]]{.endnote} and a footnote[^1].

Another paragraph with another endnote[[^E2]]{.endnote} and another footnote[^2].

[^1]: First footnote.

[^2]: Second footnote.

[^E1]: First endnote.

[^E2]: Second endnote.

On the reader side, when you specify -f docx+endnotes, you get a Span with class endnote around every Note that comes from an endnote in the docx input file.

On the writer side, when you specify -t docx+endnotes, you get an endnote in the docx output file for every Note that is embedded in a Span of class endnote.
That Span may have an identifier or other attributes, the condition is that it must have only one class: "endnote". The identifier and attributes will be discarded in the docx output, but maybe they could be useful for other output formats.

Extending endnotes extension to other formats

I found that putting this feature behind an extension that is disabled by default is a good way to ensure that no current workflow is going to be disrupted.

This convention could be extended to other formats as well (always disabled by default). The ones I thought of are:

• ConTeXt: the \endnote macro is already there, it should be easy
• ODT: similar to DOCX, but I did not inspect Pandoc's code
• HTML: since footnotes are already at the end, it should not make such a difference, but who knows?
• ICML: they should be simulated, because AFAIK InDesign does not support footnotes+endnotes both in the same document
• markdown: see below

Endnotes support in markdown

The example above (let's call it test.md) has been crafted so that footnotes and endnotes are clearly distinguishable.
Which is not the case when you make a roundtrip conversion like:

pandoc -f markdown -t markdown test.md

which gives this output:

A paragraph with an endnote[[^1]]{.endnote} and a footnote[^2].

Another paragraph with another endnote[[^3]]{.endnote} and another
footnote[^4].

[^1]: First endnote.

[^2]: First footnote.

[^3]: Second endnote.

[^4]: Second footnote.

Endnotes are still there when you export to docx+endnotes, but the markdown source is less clear.

I'd like to make the markdown format aware of the endnotes extension, so that you could use this further convention:

A paragraph with an endnote[^EN1] and a footnote[^1].

Another paragraph with another endnote[^EN2] and another footnote[^2].

[^1]: First footnote.

[^2]: Second footnote.

[^EN1]: First endnote.

[^EN2]: Second endnote.

where [EN1] becomes a synonym of [[EN1]]{.endnote} when the extension endnotes is active.
So when you convert with -f markdown+endnotes, internally the AST would be the same of the first example, with Spans of class endnote around all the Notes that in markdown have been identified with a marker that starts with a certain prefix ("EN" in this example).

On the markdown writer side, the endnotes extension would produce an output where:

• all the endnotes' markers have a common prefix ("EN" in the example)
• all the footnotes' texts are listed before all the endnotes' ones, so that you keep the distinction

Please note that when the endnotes extension is not enabled, everything still works, because all the notes are still Notes, they are simply all footnotes, while the surrounding Spans would usually go away or be transparent in many conversions to different formats.

A further choice is whether the prefix should be hardcoded or set with an option.

[massifrg]

Endnotes support for markdown, following the conventions and syntax illustrated above, has been added in #11508.
There's also a new option to set the endnotes' references prefix: --endnotes-prefix.

[iandol]

Thank you for your hard work, this has been a long requested feature for academic writers (see #1425 from 2014 and #4041 from 2017 for examples). What would happen to this if we finally get an AST change for different types of notes?

[massifrg]

Thanks @iandol for pointing me to the issues on endnotes, as I looked into discussions and PRs but not into issues!

As @jgm said here, changing the AST is painful, and it may disrupt existing workflows (disclaimer: being the author of an editor of the pandoc AST, I have one more reason not to change the AST 😁 )

That's why I went for a convention within the existing AST from the get-go, keeping the custom-style convention as a model.
This is the convention: "an endnote is just a Note embedded in a Span with the class 'endnote'".

This paragraph has an endnote[[^1]]{.endnote}.

[^1]: This is the endnote text.

Then I realized that an extension -- endnotes, disabled by default -- would have not disrupted any existing workflow, leaving every format reader and writer free to recognize it and act consequently.

Then came the idea of a prefix for endnotes, which -- looking into the issues you reported -- is not original at all.

Now you can write:

pandoc --endnotes-prefix=e -f markdown+endnotes -t markdown
This paragraph has an endnote[^e1].

[^e1]: This is the endnote text.
^D

and get the same AST of the previous markdown snippet.

[massifrg]

Doubts and possible flaws of this approach:

• computational cost: even when the endnotes extension is disabled, there are places in the code
  where its activation is checked; the frequency is tied to the number of Span inlines in a document,
  because we must check if the extension is enabled and if there's an "endnote" class

• corner cases: what to do when a Span around a Note has an "endnote" class and other information:
  i.e. non empty identifier, attributes or other classes; the current implementation keeps identifier and attributes,
  but misses the matching if there are some other classes: the condition, currently, is that there must be only
  one class and its value is "endnote";
  a similar model is the one for highlighted (marked) text, for which the condition is: a Span with an empty identifier,
  only one class with the "mark" value, and no attributes

• interaction between --endnotes-prefix and --id-prefix; in the current implementation,
  the former is put before the latter (only when endnotes extension is enabled
  and --id-prefix is set):

  pandoc --endnotes-prefix=EN --id-prefix=base -f markdown -t markdown+endnotes
  This paragraph has an endnote[[^1]]{.endnote}.
  
  [^1]: This is the endnote text.
  ^D
  This paragraph has an endnote[^ENbase1].
  
  [^ENbase1]: This is the endnote text.
  

  If the consensus is for baseEN1 instead of ENbase1, we need a small change here.
  Since there's no --id-prefix on the reader side, it would change the way to detect endnotes again:

  # current case: endnotes prefix before id prefix
  pandoc --endnotes-prefix=EN -f markdown+endnotes ...
  # alternative case: id prefix before endnotes prefix
  pandoc --endnotes-prefix=baseEN -f markdown+endnotes ...
  

• I know there's an interaction between citeproc and notes: someone with a better knowledge
  on this topic is needed to test the interaction with the convention used for endnotes

• I've set the default endnotes prefix to "EN": suggestions are welcome here, before the PR
  gets eventually merged in main

• make some other formats aware of the endnotes extension:

  • ConTeXt (only writer): it seems easy, I'm confident I can do it
  • LaTeX: maybe easier on the writer side, but I would leave this to someone else with a better LaTeX knowledge
  • ICML: it looks like there's no native support for both footnotes and endnotes in the same document,
    so endnotes should be emulated, in a way similar to footnotes in HTML; maybe it's matter for a filter
  • ODT: it may be similar to DOCX, but I need to inspect it and it's low priority for me now DONE
  • HTML: it's likely not necessary, but the endnotes already present in a document would pollute
    the resulting HTML with <span class="endnote"> elements around the footnotes refs links

[massifrg]

I nearly finished the endnotes support for ODT, because it looked easier than DOCX.

The writer is ready, while in the reader I currently can't figure out how to check a condition here to switch between note and endnote, two function that create a Note [...] or a Span ("", ["endnote"], []) [Note [...]].

Here's the function (lines 800-803):

read_note        :: InlineMatcher
read_note         = matchingElement NsText "note"
                    $ liftA (if endnotesActive then endnote else note)
                    $ matchChildContent' [ read_note_body ]

That function is called when an element like this (endnote) is parsed:

<text:note text:id="ftn1" text:note-class="endnote">
  <text:note-citation>i</text:note-citation>
  <text:note-body>
    <text:p text:style-name="P2">Endnote text.</text:p>
  </text:note-body>
</text:note>

or like this (footnote):

<text:note text:id="ftn0" text:note-class="footnote">
  <text:note-citation>1</text:note-citation>
  <text:note-body>
    <text:p text:style-name="P3">Footnote text.</text:p>
  </text:note-body>
</text:note>

endnotesActive is True when the readEndnotes field of ReaderState is True and the text:note-class attribute of the current Element in the xml parser state is "endnote".
Both the reader and the xml parser states are available in that function, but I don't know how to get them, because the underlying Arrow abstraction in the code is too hard for my current haskell knowledge.
I currently can't make the haskell types match.

[massifrg]

I found my way through the haskell types and monads of the ODT reader, now endnotes support for ODT is complete (both reader an writer).

[tajmone]

@massifrg : I strongly suggest that you consider inverting the default note type with your extension: i.e. that unmarked notes should be treated as endnotes, and change the class from .endnote to .footnote, and the sugar-marker from EN to FN so that footnotes require the special syntax rather than endnotes. In my experience in digital publications, the average ratio between footnotes and endnotes (aka Reference notes) is 5 to 50—i.e. you'll find on average around 5 footnotes and 50 endnotes per chapter, simply because bibliographic reference are usually provided for every quotation, whereas comments to the text (footnotes) are more rare.

@massifrg wrote:

HTML: since footnotes are already at the end, it should not make such a difference, but who knows?

well, if endnotes and footnotes are treated alike they could break the endnotes, depending on how they are mixed together. Endnotes are usually Reference notes (i.e. bibliographic notes), so they make abundant use of "Ibid." to reference a previous reference to a same work, but if conflating footnotes and endnotes together results in a footnote ending up between the "Ibid." reference and it's previous note, then the reference is broken. In short, if all endnotes are grouped together, in their order of appearance, *after* the footnotes, then everything should be fine; but if they are all mixed together, in their order of appearance, then endnotes are likely to break.

[massifrg]

Thanks @tajmone for your feedback!

@massifrg : I strongly suggest that you consider inverting the default note type with your extension: i.e. that unmarked notes should be treated as endnotes, and change the class from .endnote to .footnote, and the sugar-marker from EN to FN so that footnotes require the special syntax rather than endnotes. In my experience in digital publications, the average ratio between footnotes and endnotes (aka Reference notes) is 5 to 50—i.e. you'll find on average around 5 footnotes and 50 endnotes per chapter, simply because bibliographic reference are usually provided for every quotation, whereas comments to the text (footnotes) are more rare.

Also in my experience endnotes are likely the quantitative winners, but someone else could argue that he's seen nothing but footnotes (while I'm writing this I'm hearing jgm's voice telling me: "there are [just] notes", wherever you put them 😁 ).
I would keep that, because footnotes are usually the first kind of notes that come to mind, or at least many softwares support first.

The main reason that pushed me to try to code it in pandoc's haskell is that you can't provide support for endnotes in DOCX with ordinary filters, because endnotes, as well as footnotes, are not in the main document file inside the DOCX (zip) package. So no way to access them with a filter.

The prefix trick in markdown is just an answer to: "Ok, I polluted markdown with those [ ]{.endnote}, is there a way to mitigate that (without disrupting anyone's workflow)?"

The ODT support is just a challenge, since I realized that (foot|end)notes are coded in the main XML document, so -- at least on the writer side -- you can differentiate notes via filters and RawInlines.

well, if endnotes and footnotes are treated alike they could break the endnotes, depending on how they are mixed together.

This is feasible with a lua filter, intercepting Note inlines and Spans (in case you use the [ ]{.endnote} convention): you prevent Pandoc from converting Notes to HTML the usual way, and you do it with RawInlines, accumulating the text notes in two different pandoc.Lists, one for footnotes and the other one for endnotes. Something like this:

local footnotes = pandoc.List()
local endnotes = pandoc.List()
local collect_notestexts_and_insert_refs = {
  traverse: "topdown",
  Note = function(note)
    -- add to footnotes List and insert a reference with RawInlines
  end,
  Span = function(span)
    -- check if span.contents is made just of a Note; if so, then add to endnotes List and insert a reference
  end
}
local append_notes = {
  Pandoc = function(doc)
    -- extend doc.blocks with footnotes and endnotes texts
  end
}
return { collect_notestexts_and_insert_refs, append_notes }

In case you use a prefix text in notes' contents to differentiate, it's a bit more complex, but still feasible.

[tajmone]

Indeed, that's the approach I've taken with the filter: use two independent lists. My concern was mostly about when users don't enable the endonotes extensions and want to just keep all notes as notes (e.g. because converting to Obsidian markdown); in similar cases, if pandoc just emits the notes in their order of appearance, the risk is breaking some endnotes references, but on the other hand if pandoc were to handle all the footnotes first, followed by the endonotes (i.e. both as normal notes), then the problem would be the notes numbering, e.g. note 3 could be followed by note 62 because the latter was originally meant to be an endnotes, so it has a higher note number compared to the footnote. Most likely, any document intended to have dual notes will not be buildable without problems if the endnotes extension is not enabled. But having some options on how to handle the normalization of endonotes might be nice.