Standard YAML frontmatter syntax causes lists to not be detected

[nathanlesage]

Initial checklist

• I read the support docs
• I read the contributing guide
• I agree to follow the code of conduct
• I searched issues and couldn’t find anything (or linked relevant results below)

Affected packages and versions

micromark-extension-frontmatter v1.0.0

Link to runnable example

https://codesandbox.io/s/hopeful-matan-nybskz?file=%2Fsrc%2Findex.js

Steps to reproduce

I have a workload where I need to transform Markdown documents that can also include frontmatters to a syntax tree, using parse() on the remark processor.

It breaks when that said YAML frontmatter uses Jekyll-style delimiters (three dashes both on top and on the bottom of the frontmatter). It works, however, when I use Pandoc-style delimiters (three dashes on top, three dots on the bottom). Usage of the plugin then disables the processor's ability to detect lists properly.

Here are the steps to reproduce. Above I have linked a sandbox that is already properly set up. In there, simply exchange the end-fence for the frontmatter between dashes and dots and observe the difference in produced syntax trees.

Processor setup

export function md2ast (markdown: string): Root {
  return remark()
    .use(remarkFrontmatter, [
      // Either Pandoc-style frontmatters ...
      { type: 'yaml', fence: { open: '---', close: '...' } },
      // ... or Jekyll/Static site generators-style frontmatters.
      { type: 'yaml', fence: { open: '---', close: '---' } }
    ])
    .use(remarkMath)
    .parse(markdown)
}

Note that while I am being verbose in explicitly stating the delimiters, exchanging the second definition to simply yaml does not change the effect.

Test document

Use this document and run it through said pipeline. You will notice that, when you use three dashes to end the frontmatter, the list is not correctly detected, whereas, when you exchange that with three dots (Pandoc-style frontmatter), it will be correctly detected.

---
title: "The Devil is in the Details: Ethical Pitfalls in the Sociological use of NLP techniques"
date: 2022-12-05
id: 20221205151411
author: Hendrik Erz
---

# Export Link Removal

Export this file into any format to test out the corresponding LUA filter.

* History of NLP in Sociology (from Mosteller and Wallace to today)
* What methods are in use? (three types: Bayes/simple such as Logistic Regression; Machine Learning such as LDA/random forests; deep learning such as LSTM/BERT)
* What are they being used for?
    * This is a worngly written second-indended word
* Where do these methods come from?
* Are there already ethical notes around? Or don’t they care?

Expected behavior

Correct behavior with Pandoc-style frontmatter

If you exchange the three dashes with three dots in the test-document, it works as expected.

This is the correct Syntax Tree:

{
  "type": "root",
  "children": [
    {
      "type": "yaml",
      "value": "title: \"The Devil is in the Details: Ethical Pitfalls in the Sociological use of NLP techniques\"\ndate: 2022-12-05\nid: 20221205151411\nauthor: Hendrik Erz",
      "position": {
        "start": { "line": 1, "column": 1, "offset": 0 },
        "end": { "line": 6, "column": 4, "offset": 160 }
      }
    },
    {
      "type": "heading",
      "depth": 1,
      "children": [
        {
          "type": "text",
          "value": "Export Link Removal",
          "position": {
            "start": { "line": 8, "column": 3, "offset": 164 },
            "end": { "line": 8, "column": 22, "offset": 183 }
          }
        }
      ],
      "position": {
        "start": { "line": 8, "column": 1, "offset": 162 },
        "end": { "line": 8, "column": 22, "offset": 183 }
      }
    },
    {
      "type": "paragraph",
      "children": [
        {
          "type": "text",
          "value": "Export this file into any format to test out the corresponding LUA filter.",
          "position": {
            "start": { "line": 10, "column": 1, "offset": 185 },
            "end": { "line": 10, "column": 75, "offset": 259 }
          }
        }
      ],
      "position": {
        "start": { "line": 10, "column": 1, "offset": 185 },
        "end": { "line": 10, "column": 75, "offset": 259 }
      }
    },
    {
      "type": "list",
      "ordered": false,
      "start": null,
      "spread": false,
      "children": [
        {
          "type": "listItem",
          "spread": false,
          "checked": null,
          "children": [
            {
              "type": "paragraph",
              "children": [
                {
                  "type": "text",
                  "value": "History of NLP in Sociology (from Mosteller and Wallace to today)",
                  "position": {
                    "start": { "line": 12, "column": 3, "offset": 263 },
                    "end": { "line": 12, "column": 68, "offset": 328 }
                  }
                }
              ],
              "position": {
                "start": { "line": 12, "column": 3, "offset": 263 },
                "end": { "line": 12, "column": 68, "offset": 328 }
              }
            }
          ],
          "position": {
            "start": { "line": 12, "column": 1, "offset": 261 },
            "end": { "line": 12, "column": 68, "offset": 328 }
          }
        },
        {
          "type": "listItem",
          "spread": false,
          "checked": null,
          "children": [
            {
              "type": "paragraph",
              "children": [
                {
                  "type": "text",
                  "value": "What methods are in use? (three types: Bayes/simple such as Logistic Regression; Machine Learning such as LDA/random forests; deep learning such as LSTM/BERT)",
                  "position": {
                    "start": { "line": 13, "column": 3, "offset": 331 },
                    "end": { "line": 13, "column": 161, "offset": 489 }
                  }
                }
              ],
              "position": {
                "start": { "line": 13, "column": 3, "offset": 331 },
                "end": { "line": 13, "column": 161, "offset": 489 }
              }
            }
          ],
          "position": {
            "start": { "line": 13, "column": 1, "offset": 329 },
            "end": { "line": 13, "column": 161, "offset": 489 }
          }
        },
        {
          "type": "listItem",
          "spread": false,
          "checked": null,
          "children": [
            {
              "type": "paragraph",
              "children": [
                {
                  "type": "text",
                  "value": "What are they being used for?",
                  "position": {
                    "start": { "line": 14, "column": 3, "offset": 492 },
                    "end": { "line": 14, "column": 32, "offset": 521 }
                  }
                }
              ],
              "position": {
                "start": { "line": 14, "column": 3, "offset": 492 },
                "end": { "line": 14, "column": 32, "offset": 521 }
              }
            },
            {
              "type": "list",
              "ordered": false,
              "start": null,
              "spread": false,
              "children": [
                {
                  "type": "listItem",
                  "spread": false,
                  "checked": null,
                  "children": [
                    {
                      "type": "paragraph",
                      "children": [
                        {
                          "type": "text",
                          "value": "This is a worngly written second-indended word",
                          "position": {
                            "start": { "line": 15, "column": 7, "offset": 528 },
                            "end": { "line": 15, "column": 53, "offset": 574 }
                          }
                        }
                      ],
                      "position": {
                        "start": { "line": 15, "column": 7, "offset": 528 },
                        "end": { "line": 15, "column": 53, "offset": 574 }
                      }
                    }
                  ],
                  "position": {
                    "start": { "line": 15, "column": 5, "offset": 526 },
                    "end": { "line": 15, "column": 53, "offset": 574 }
                  }
                }
              ],
              "position": {
                "start": { "line": 15, "column": 5, "offset": 526 },
                "end": { "line": 15, "column": 53, "offset": 574 }
              }
            }
          ],
          "position": {
            "start": { "line": 14, "column": 1, "offset": 490 },
            "end": { "line": 15, "column": 53, "offset": 574 }
          }
        },
        {
          "type": "listItem",
          "spread": false,
          "checked": null,
          "children": [
            {
              "type": "paragraph",
              "children": [
                {
                  "type": "text",
                  "value": "Where do these methods come from?",
                  "position": {
                    "start": { "line": 16, "column": 3, "offset": 577 },
                    "end": { "line": 16, "column": 36, "offset": 610 }
                  }
                }
              ],
              "position": {
                "start": { "line": 16, "column": 3, "offset": 577 },
                "end": { "line": 16, "column": 36, "offset": 610 }
              }
            }
          ],
          "position": {
            "start": { "line": 16, "column": 1, "offset": 575 },
            "end": { "line": 16, "column": 36, "offset": 610 }
          }
        },
        {
          "type": "listItem",
          "spread": false,
          "checked": null,
          "children": [
            {
              "type": "paragraph",
              "children": [
                {
                  "type": "text",
                  "value": "Are there already ethical notes around? Or don’t they care?",
                  "position": {
                    "start": { "line": 17, "column": 3, "offset": 613 },
                    "end": { "line": 17, "column": 62, "offset": 672 }
                  }
                }
              ],
              "position": {
                "start": { "line": 17, "column": 3, "offset": 613 },
                "end": { "line": 17, "column": 62, "offset": 672 }
              }
            }
          ],
          "position": {
            "start": { "line": 17, "column": 1, "offset": 611 },
            "end": { "line": 17, "column": 62, "offset": 672 }
          }
        }
      ],
      "position": {
        "start": { "line": 12, "column": 1, "offset": 261 },
        "end": { "line": 17, "column": 62, "offset": 672 }
      }
    }
  ],
  "position": {
    "start": { "line": 1, "column": 1, "offset": 0 },
    "end": { "line": 18, "column": 1, "offset": 673 }
  }
}

Actual behavior

Incorrect behavior with Jekyll-style frontmatter

If you just run the above test document (with three dashes), it produces a wrong syntax tree. Observe how it does not detect the list properly.

{
  "type": "root",
  "children": [
    {
      "type": "yaml",
      "value": "title: \"The Devil is in the Details: Ethical Pitfalls in the Sociological use of NLP techniques\"\ndate: 2022-12-05\nid: 20221205151411\nauthor: Hendrik Erz",
      "position": {
        "start": { "line": 1, "column": 1, "offset": 0 },
        "end": { "line": 6, "column": 4, "offset": 160 }
      }
    },
    {
      "type": "heading",
      "depth": 1,
      "children": [
        {
          "type": "text",
          "value": "Export Link Removal",
          "position": {
            "start": { "line": 8, "column": 3, "offset": 164 },
            "end": { "line": 8, "column": 22, "offset": 183 }
          }
        }
      ],
      "position": {
        "start": { "line": 8, "column": 1, "offset": 162 },
        "end": { "line": 8, "column": 22, "offset": 183 }
      }
    },
    {
      "type": "paragraph",
      "children": [
        {
          "type": "text",
          "value": "Export this file into any format to test out the corresponding LUA filter.",
          "position": {
            "start": { "line": 10, "column": 1, "offset": 185 },
            "end": { "line": 10, "column": 75, "offset": 259 }
          }
        }
      ],
      "position": {
        "start": { "line": 10, "column": 1, "offset": 185 },
        "end": { "line": 10, "column": 75, "offset": 259 }
      }
    },
    {
      "type": "paragraph",
      "children": [
        {
          "type": "text",
          "value": "* History of NLP in Sociology (from Mosteller and Wallace to today)\n* What methods are in use? (three types: Bayes/simple such as Logistic Regression; Machine Learning such as LDA/random forests; deep learning such as LSTM/BERT)\n* What are they being used for?\n* This is a worngly written second-indended word\n* Where do these methods come from?\n* Are there already ethical notes around? Or don’t they care?",
          "position": {
            "start": { "line": 12, "column": 1, "offset": 261 },
            "end": { "line": 17, "column": 62, "offset": 672 }
          }
        }
      ],
      "position": {
        "start": { "line": 12, "column": 1, "offset": 261 },
        "end": { "line": 17, "column": 62, "offset": 672 }
      }
    }
  ],
  "position": {
    "start": { "line": 1, "column": 1, "offset": 0 },
    "end": { "line": 18, "column": 1, "offset": 673 }
  }
}

Runtime

Node v16, Other (please specify in steps to reproduce)

Package manager

yarn v1

OS

macOS

Build and bundle tools

Webpack

[wooorm]

So first, for the code sandbox:

Frontmatter starts at the first line. You can test this on GH: adding blank lines before the YAML doesn’t work. Start with:

const sourceMarkdown = `---

…and it works.

[wooorm]

It breaks when that said YAML frontmatter uses Jekyll-style delimiters (three dashes both on top and on the bottom of the frontmatter)

What doesn’t work? I believe it works. That’s the default. That’s what 1.6m people per month use the plugin for. And it’s well tested

[wooorm]

Here’s a reduced test case:

import {fromMarkdown} from 'mdast-util-from-markdown'
import {frontmatter} from 'micromark-extension-frontmatter'
import {frontmatterFromMarkdown} from 'mdast-util-frontmatter'

const sourceMarkdown = `---
title: a

* b
`;

const tree = fromMarkdown(sourceMarkdown, {
  extensions: [frontmatter()],
  mdastExtensions: [frontmatterFromMarkdown()]
})

console.log(tree) // Contains a paragraph with `* b` as text, not a list.

The reason is that this looks a lot like frontmatter. So it continues on checking all those lines for the ending. When doing that, it prevents the lists from being checked. Afterwards, it can’t do those lists again.

[github-actions]

Hi! This was marked as ready to be worked on! Note that while this is ready to be worked on, nothing is said about priority: it may take a while for this to be solved.

Is this something you can and want to work on?

Team: please use the area/* (to describe the scope of the change), platform/* (if this is related to a specific one), and semver/* and type/* labels to annotate this. If this is first-timers friendly, add good first issue and if this could use help, add help wanted.

[nathanlesage]

Thank you for your swift reply! You are right that I accidentally added a newline there. However, when I remove that, the list is still not properly being detected. Here's a screenshot from the updated sandbox:

What doesn’t work? I believe it works. That’s the default. That’s what 1.6m people per month use the plugin for. And it’s well tested

I am sorry if I came across as ungrateful or anything — I am a big fan of your work and appreciate everything you do. I did some extensive checking this morning to find the bug, and ended up finding this oddity. I am, however, not sure what kind of interaction effect is (I believe that each plugin itself works fine, absolutely, but since we have this weird bug, I suspect some interaction – that's also why I'm not sure if this emanates from here, or from the remark plugin that wraps this one…? I am thankful for any guidance in this regard.)

Regarding your reduced case, what is the intention in leaving out the ending fence of the frontmatter here? In other words, why did you not write the following?

---
title: a
---

* b

---

One additional thing I noted, but which I believe can be fixed when this bug is sorted out, is that if the list is not being properly detected, the parser "eats" the indentation spaces in front of the second-order list item. You may note that there is this worngly spelled word in the indentation. The reason is that I'm using the syntax tree to run a spell checker, and I noted that the spaces in front of the list just get swallowed, shifting the offsets of the text nodes into a wrong position by that amount. I did not initially report that because I believe it is just a fallout effect of whatever this is.

I greatly appreciate any guidance in this regard. Thank you!

[wooorm]

am sorry if I came across as ungrateful or anything

No need, no worries, you didn’t come across as such. See my later response for a reduced case. You are correct, there is a bug.

Regarding your reduced case, what is the intention in leaving out the ending fence of the frontmatter here? In other words, why did you not write the following?

Because that’s exactly what’s going on here: if an opening but no closing fence is found, lists don’t work. You had --- and ... in your first case. Which, when starting with --- but without an ending of ..., shows that exact bug. But your second case (start and end ---) masked what was happening. Because some frontmatter was parsed.

---

For the second bug, whitespace being eaten, I think it’s unrelated to this, probably. It’s because whitespace in paragraphs is removed. E.g.:

asd
                                   whatever

That positional info might be lost somewhere though. As the text node is asd\nwhatever for that. But also this markdown would result in the exact same text node:

> * asd
> whatever

Maybe some positional info is lost somewhere

[nathanlesage]

Ahhhh, I see what you mean, yes! Thank you for the clarification!

[nathanlesage]

Hey @wooorm, just as an update: I have now figured out that the fault in the options also prevents the Gfm parser (remark-gfm) from detecting footnotes (possibly other elements as well). It works fine if I just comment out the corresponding config option { type: 'yaml', fence: { open: '---', close: '...' } }.

[wooorm]

Yep, block quotes, lists, and footnote definitions! All “containers” that fail on this :(
Will be maybe a bit less than 2 months before I can get to it unfortunately.