RFC: Represent citation locators structurally in the Pandoc AST

[dereuromark]

Describe your proposed improvement and the problem it solves.

Pandoc currently stores a citation locator as part of citationSuffix:

[@doe, p. 33]

The AST retains roughly , p. 33, but not that page is the locator label and 33 is its value. Citeproc can parse this convention again, but filters and format bridges cannot inspect or change the locator reliably without parsing formatted inline content.

Could Citation gain optional structured locator information, such as a label and inline value? Readers could populate it when the source format provides that structure, and citeproc could continue accepting the existing suffix convention for compatibility.

This came up while mapping Carve citations, but it would also help CSL JSON, JATS, DOCX and other structured citation formats.

Describe alternatives you've considered.

We preserve the visible locator in citationSuffix and have since added private round-trip metadata carrying the typed label and value, keeping a native Cite alongside it so citeproc still processes the citation normally. That keeps output working and round-trips our own documents, but it does not give Pandoc filters an interoperable representation - a filter would have to know our private encoding to read a locator, which is the thing worth avoiding.

I realize changing pandoc-types has a high compatibility cost. I am opening this mainly to establish whether structured locators belong in the AST before attempting a concrete API proposal.

[jgm]

Aside from the cost of modifying pandoc-types, one issue is that parsing locators is localization dependent, and the localization information lives in citeproc. Otherwise I agree that this would be good.
[Actually, we already parse the locators in pandoc, in Text.Pandoc.Citeproc.Locator. So it's feasible with a pandoc-types API change.]

[jgm]

For reference, pandoc-types:

inline constructor `Cite [Citation] [Inline]`.

data Citation = Citation { citationId      :: Text
                         , citationPrefix  :: [Inline]
                         , citationSuffix  :: [Inline]
                         , citationMode    :: CitationMode
                         , citationNoteNum :: Int
                         , citationHash    :: Int
                         }
                deriving (Show, Eq, Read, Typeable, Data, Generic)

instance Ord Citation where
    compare = comparing citationHash

data CitationMode = AuthorInText | SuppressAuthor | NormalCitation
                    deriving (Show, Eq, Ord, Read, Typeable, Data, Generic)

citeproc:

data Citation a =
  Citation { citationId            :: Maybe Text
           , citationResetPosition :: Bool -- ^ Reset the position information
             -- that determines whether citation items are "subsequent,"
             -- prior to processing this citation.
             -- Note that this affects all subsequent citations, not just this
             -- one.
           , citationNoteNumber    :: Maybe Int
           , citationPrefix        :: Maybe a
           , citationSuffix        :: Maybe a
           , citationItems         :: [CitationItem a]
           }
  deriving (Show, Eq, Ord)

data CitationItem a =
  CitationItem
  { citationItemId             :: ItemId
  , citationItemLabel          :: Maybe Text
  , citationItemLocator        :: Maybe Text
  , citationItemType           :: CitationItemType
  , citationItemPrefix         :: Maybe a
  , citationItemSuffix         :: Maybe a
  , citationItemData           :: Maybe (Reference a)
  } deriving (Show, Eq, Ord)

data CitationItemType =
    AuthorOnly      -- ^ e.g., Smith
  | SuppressAuthor  -- ^ e.g., (2000, p. 30)
  | NormalCite      -- ^ e.g., (Smith 2000, p. 30)
  deriving (Show, Eq, Ord)

The interface between them is at https://github.com/jgm/pandoc/blob/main/src/Text/Pandoc/Citeproc.hs#L294-L364

[dereuromark]

Cool.
Would locatorRaw be worth a third field? It is what makes a source-to-source round trip
exact when the source spelled the label in a way the locale map normalizes, and it is
already computed. I would understand leaving it out.

[jgm]

The sensible changes to pandoc Citation would be:

• Eliminate citationHash field; it is a holdover from the old pandoc-citeproc.
• Add citationLocator and citationLocatorLabel (both Maybe Text).

[jgm]

I would not be inclined to add locatorRaw. The pandoc AST is meant to be abstract, and it isn't designed for exact source-to-source round trips. (For that we'd need to add similar things all over the place.)

[dereuromark]

That shape works for me, and citationLocatorLabel is a better name than the citationLabel I had in mind.

Does the suffix keep the locator text?
It cannot, I think: parseLocator strips it today, so a reader that populated both fields and also left , p. 33 in
citationSuffix would print it twice. Worth stating that when the fields are set, the suffix holds only what follows the locator.

[jgm]

Thinking how it would work on the writer side: given a label and locator, how to produce a Markdown citation. We'd need to find the abbreviation for the label that is appropriate for the locale. Finding the locale requires having the CSL style, but we should have access to that in the writer (as it's in Meta). We have toLocatorMap which creates a mapping from abbrevations -> locator label names; we'd have to reverse this mapping and figure out what to do when multiple strings map to the same label name.

An added complication is that the labels come in "single" and "multiple" forms; citeproc figures out the relevant form from the locator itself. We'd have to reproduce this logic in the markdown writer to figure out whether to use a single form (e.g. p.) or a multiple form (pp.). This also means that merely reversing the LocatorMap wouldn't be enough, because it doesn't contain single/multiple information; we'd need a richer LocatorMap structure or a new ReverseLocatorMap or something...