can you have a table of contents with closeread?

[mm225022]

I understand that the intent of scrollytelling is for the user to scroll through the whole document, but would it be possible to have toc?
That might be helpful to jump to a section or wehen re-reading.

I tried with the build-up plots demo, and the toc only shows for the sections outside the .cr-section fenced div:

---
title: Build up plots
subtitle: "If your plot is too complicated, try introducing elements one at a time."
format:
  closeread-html:
    code-fold: true
    cr-section:
      layout: overlay-left
toc: true
knitr:
  opts_chunk: 
    dev.args:
      bg: transparent
---

### Intro

In this demo, we'll recreate a plot from the documentation of Allison Horst's [`{palmerpenguins}`](https://allisonhorst.github.io/palmerpenguins/) R package, which provides access to penguin data from Dr. Kristen Gorman and the Palmer Station in Antarctica.

```{r}
#| label: setup
#| echo: true
#| output: false
library(dplyr)
library(ggplot2)
library(palmerpenguins)

penguins |>
  filter(
    !is.na(species),
    !is.na(flipper_length_mm),
    !is.na(body_mass_g)) ->
penguins_forplot
```

#### Final Plot

The plot looks like this:

```{r}
#| label: original-plot
#| echo: true
ggplot(penguins_forplot) +
  aes(
    x = flipper_length_mm,
    y = body_mass_g) +
  geom_point(
    aes(
      colour = species,
      shape = species)) +
  theme_minimal() +
  theme(
    legend.position = "inside",
    legend.position.inside = c(0.9, 0.15)) +
  labs(
    x = "Flipper length (mm)",
    y = "Body mass (g)")
```

There's a bit going on in this dataset! What if we built it up one step at a time instead?

The key to seamlessly blending graphics is making sure they're exactly the same size.

\
\

::::{.cr-section}

### Hide the data

To start, let's hide all of the data and add some explanatory text annotations. @cr-penguins-blank

:::{#cr-penguins-blank}
```{r}
#| label: blank-plot
ggplot(penguins_forplot) +
  aes(
    x = flipper_length_mm,
    y = body_mass_g) +
  geom_point(
    aes(
      colour = species,
      shape = species),
    alpha = 0) +
  annotate("text", x = Inf, y = -Inf,
    label = "Penguins with long flippers →",
    hjust = 1, vjust = -0.5) +
  annotate("text", x = -Inf, y = Inf,
    label = "↑\nHeavy\npenguins",
    hjust = 0, vjust = 1) +
  theme_minimal() +
  guides(colour = "none", shape = "none") +
  theme(
    legend.position = "inside",
    legend.position.inside = c(0.9, 0.15)) +
  labs(
    x = "Flipper length (mm)",
    y = "Body mass (g)")
```

:::

### Reveal the data

Now let's reveal the data... but without talking about the species just yet. \
\
We can already see that heavier penguins tend to have longer flippers. @cr-penguins-nospecies

:::{#cr-penguins-nospecies}
```{r}
#| label: no-species-plot
ggplot(penguins_forplot) +
  aes(
    x = flipper_length_mm,
    y = body_mass_g) +
  geom_point(
    aes(
      colour = species,
      shape = species)) +
  scale_colour_manual(values = c(
    "Adelie" = "#000000",
    "Chinstrap" = "#000000",
    "Gentoo" = "#000000"
  )) +
  scale_shape_manual(values = c(
    "Adelie" = 16,
    "Chinstrap" = 16,
    "Gentoo" = 16
  )) +
  guides(colour = "none", shape = "none") +
  theme_minimal() +
  theme(
    legend.position = "inside",
    legend.position.inside = c(0.9, 0.15)) +
  labs(
    x = "Flipper length (mm)",
    y = "Body mass (g)")
```
:::

### Adding color and shape

By adding colour and shape, we can now move to our plot's second message: \
\
Some species are heavier and larger than others! @cr-penguins-species

:::{#cr-penguins-species}
```{r}
#| label: species-plot
ggplot(penguins_forplot) +
  aes(
    x = flipper_length_mm,
    y = body_mass_g) +
  geom_point(
    aes(
      colour = species,
      shape = species),
  ) +
  theme_minimal() +
  theme(
    legend.position = "inside",
    legend.position.inside = c(0.9, 0.15)) +
  labs(
    x = "Flipper length (mm)",
    y = "Body mass (g)")
```
:::

::::

\
\

And there we have it!

## Things to help sell the illusion

There are a few things to keep in mind if you want a "built-up" plot to blend seamlessly.

When one plot replaces another, it's crucial that things like your axes don't move. By default, `ggplot2` will calculate axes based on the data you're plotting, so if your data changes - for example, because you're not ready to show all of it - the axes will move, breaking the illusion.

One way to get around this to "pre-calculate" the domain of your axes and specify those limits (with `xlim()` and `ylim()` in `ggplot2`, or using the `limits` argument in a scale function).

Another way around it to include all of your plot layers in every step and to simply hide the ones you don't want to show with zero opacity (in `ggplot2`, this is `alpha = 0` argument, or providing a colour with zero opacity).

Your axes might also move because one step has a legend and another doesn't - for example, when we show a step where all of the points are the same colour and shape.

Again, there are two ways around it. The easiest, which we've used, is to put the legend _inside_ the plot space - that way, `ggplot2` doesn't reserve any space for it.

The other way is to still map colour and shape even for the step where you want all the points to be the same, but to use `scale_[color/shape]_manual()` to set all of the species to black circles. You may still want to manually hide your legend in this case.

```{=html}
<!-- some quick style customisation: see https://closeread.dev/guide/styling -->
<style>
  .cr-section.overlay-left {
    background-color: aliceblue;
  }

  .cr-section.overlay-left .narrative-col .narrative {
    font-size: 1.15em;
    background-color: #11111199;
    color: #f1f1f1;
    padding: 0.6em 0.8em;
    border-radius: 0;
    backdrop-filter: blur(10px);
  }
</style>
```

[andrewpbray]

I noticed this behavior when we were first building Closeread and was puzzled by it, so thanks for the issue!

I've tinkered with this a bit and I can't quite figure out. Here is the document that I've been using to troubleshoot:

---
format: markdown
toc: true
filters: 
  - _extensions/qmd-lab/closeread/closeread.lua
---

## Quarto

:::{.cr-section}

## Hello

:::

:::{.class1 .class2 .class3}
:::{.class4}
:::{.class5}

## World {#world .narrative}

:::
:::
:::

(I find the markdown output is easier to read than HTML). This renders to the following (regardless of when you run the filter):

-   [Quarto](#quarto){#toc-quarto}
-   [World](#world){#toc-world}

## Quarto

::::::: {.cr-section .cr-column-screen .sidebar-left}
:::: narrative-col
::: trigger
## Hello {#hello .narrative}
:::
::::

:::: sticky-col
::: sticky-col-stack
:::
::::
:::::::

::::: {.class1 .class2 .class3}
:::: class4
::: class5
## World {#world .narrative}
:::
::::
:::::

This example is designed so that the first header is in the body of the doc like normal, the second one is inside a .cr-section and therefore processed by our lua filter, and the third is in the same AST structure as the filter-processed section (triply nested with attributes added). This tells me that it's not the structure of the AST that our filter creates that's the problem. But I'm not sure what is.

One thing I can't figure out is when in the render process the TOC is written. I've written quarto.log.output() statements into closeread.lua to peek at what the AST looks like at various points. I can't find evidence of the contents of the TOC in either the Meta or the Blocks of the AST, which suggests either a) that it's being written to an external file before our filter or b) it gets written after our filter runs. For both circumstances, I don't understand why the example TOC picks up World but not Hello.

I'll keep noodling over this... 🤔

[jimjam-slam]

I haven't had a play with the example yet, but I wonder if Quarto expects hn elements to be either directly inside the main content or within a section (which it creates when it encounters headers in the Markdown input). Although you can put headers inside column-* divs without a problem afaik...

[andrewpbray]

The creation of section only happens in HTML formats, right? Would that pertain to the format: markdown example here?

I had another theory. If the TOC is made by pandoc running after our filter, then maybe it's is looking for attributes (like identifiers) that we scrub when we rebuild the cr-section part of the AST. Alas, this didn't pan out: it already propagates any ids or classes on ## Hello and when I modify wrap_block to also propagate attributes (which it currently shifts to the enclosing block) Hello still doesn't find it's way into the TOC.

[andrewpbray]

Maybe this is a question for the Quarto discussion: How does Quarto/Pandoc detect hn elements for use in the TOC?

[jimjam-slam]

The creation of section only happens in HTML formats, right? Would that pertain to the format: markdown example here?

Ahh, of course!

[jimjam-slam]

Maybe this is a question for the Quarto discussion: How does Quarto/Pandoc detect hn elements for use in the TOC?

I think that makes sense!

Also pasting from the Pandoc docs:

--toc[=true|false], --table-of-contents[=true|false]
Include an automatically generated table of contents (or, in the case of latex, context, docx, odt, opendocument, rst, or ms, an instruction to create one) in the output document. This option has no effect unless -s/--standalone is used, and it has no effect on man, docbook4, docbook5, or jats output.

Note that if you are producing a PDF via ms, the table of contents will appear at the beginning of the document, before the title. If you would prefer it to be at the end of the document, use the option --pdf-engine-opt=--no-toc-relocation.

--toc-depth=NUMBER
Specify the number of section levels to include in the table of contents. The default is 3 (which means that level-1, 2, and 3 headings will be listed in the contents).

[jimjam-slam]

According to this Pandoc issue (if I'm reading it correctly), by design a heading only gets added to the TOC when it's inside a fenced div if it's the only heading there and it's the first thing in the div. That would mean you could only have a single heading within a .cr-section, regardless of whether the Closeread filter ran.

I wonder if there's a way we can circumvent that? I don't love asking our users to write raw HTML blocks, let alone broken ones (which invite all sorts of invalid input)...

[andrewpbray]

Good find!

This example demonstrates the first thing there requirement:

---
format: markdown
toc: true
---

## Quarto

:::{}

apple

## World

:::

which renders to

-   [Quarto](#quarto){#toc-quarto}

## Quarto

<div>

apple

## World

</div>

[andrewpbray]

This would definitely lead to more AST bloat, but we could wrap any header directly in its own personal fenced div with no attributes?

[andrewpbray]

Uf, nvm. Being a nested div doesn't protect it from the requirements of Pandoc.

[andrewpbray]

We could always directly rewrite the pandoc-emitted HTML using either a post-render .ts script or our JS. 😬

[andrewpbray]

Okay, this notion of a fended div being an isolated environments is interesting. I hadn't realized that before but it makes sense in the context of callouts (I had assumed the co-opting of ## was a Quarto thing, not a Pandoc thing). This is very useful to know in general...

[jimjam-slam]

We could always directly rewrite the pandoc-emitted HTML using either a post-render .ts script or our JS. 😬

I wonder if there’s a way to gain access to the TOC before we shift out of Pandoc land and into Quarto HTML post-processing. Is it in the meta in filters? Can we modify it? Otherwise this may be the way to go! But yeah, 😬 Maybe the Quarto team has some ideas on other points to get involved…

[jimjam-slam]

Okay, this notion of a fended div being an isolated environments is interesting. I hadn't realized that before but it makes sense in the context of callouts (I had assumed the co-opting of ## was a Quarto thing, not a Pandoc thing). This is very useful to know in general...

I wonder if it also explains headers and the content below them generally being wrapped in section tags. Does Pandoc do that, or does Quarto do it beforehand?

[andrewpbray]

I wonder if there’s a way to gain access to the TOC before we shift out of Pandoc land and into Quarto HTML post-processing.

I couldn't find a lick of it in either Meta or Body and I tried running the filter at every stage in the process that Quarto exposes.

I wonder if it also explains headers and the content below them generally being wrapped in section tags. Does Pandoc do that, or does Quarto do it beforehand?

Let's test. Let's start with this markdown:

## Quarto

:::{}

apple

:::{.class1}

## World

:::
:::

Here's the result of quarto render to format: html, focusing just on the body part:

<section id="quarto" class="level2">
<h2 class="anchored" data-anchor-id="quarto">Quarto</h2>
<div>
<p>apple</p>
<section id="world" class="level2 class1">
<h2 class="anchored" data-anchor-id="world">World</h2>
</section>
</div>
</section>

If you instead run pandoc small-toc.qmd -o small-toc.html, you get:

<h2 id="quarto">Quarto</h2>
<div>
<p>apple</p>
<section id="world" class="class1">
<h2>World</h2>
</section>
</div>

Pandoc's behavior here is kinda surprising to me. I did some haphazard testing, and it seems like, no matter how deeply nested the divs are, its only the innermost div that pandoc will wrap in a <section> tag, and only if the first element in that div is the header (and it can be any level header). Quarto must take one more run through the doc to append those other sections.