AST for lists is not mdast compliant

[agoose77]

Description

The mdast spec for lists is that listItem contains FlowContent, ultimately Content, that contains Paragraph of Text:

console.dir(
  fromMarkdown(
    `
- This is an item
- This is also an item
- Would you believe it, this is _also_ an item!
`,
  ),
  { depth: null },
);

Produces

Details

{
  "type": "root",
  "children": [
    {
      "type": "list",
      "ordered": false,
      "start": null,
      "spread": false,
      "children": [
        {
          "type": "listItem",
          "spread": false,
          "checked": null,
          "children": [
            {
              "type": "paragraph",
              "children": [
                {
                  "type": "text",
                  "value": "This is an item",
                  "position": {
                    "start": { "line": 2, "column": 3, "offset": 3 },
                    "end": { "line": 2, "column": 18, "offset": 18 }
                  }
                }
              ],
              "position": {
                "start": { "line": 2, "column": 3, "offset": 3 },
                "end": { "line": 2, "column": 18, "offset": 18 }
              }
            }
          ],
          "position": {
            "start": { "line": 2, "column": 1, "offset": 1 },
            "end": { "line": 2, "column": 18, "offset": 18 }
          }
        },
        {
          "type": "listItem",
          "spread": false,
          "checked": null,
          "children": [
            {
              "type": "paragraph",
              "children": [
                {
                  "type": "text",
                  "value": "This is also an item",
                  "position": {
                    "start": { "line": 3, "column": 3, "offset": 21 },
                    "end": { "line": 3, "column": 23, "offset": 41 }
                  }
                }
              ],
              "position": {
                "start": { "line": 3, "column": 3, "offset": 21 },
                "end": { "line": 3, "column": 23, "offset": 41 }
              }
            }
          ],
          "position": {
            "start": { "line": 3, "column": 1, "offset": 19 },
            "end": { "line": 3, "column": 23, "offset": 41 }
          }
        },
        {
          "type": "listItem",
          "spread": false,
          "checked": null,
          "children": [
            {
              "type": "paragraph",
              "children": [
                {
                  "type": "text",
                  "value": "Would you believe it, this is ",
                  "position": {
                    "start": { "line": 4, "column": 3, "offset": 44 },
                    "end": { "line": 4, "column": 33, "offset": 74 }
                  }
                },
                {
                  "type": "emphasis",
                  "children": [
                    {
                      "type": "text",
                      "value": "also",
                      "position": {
                        "start": { "line": 4, "column": 34, "offset": 75 },
                        "end": { "line": 4, "column": 38, "offset": 79 }
                      }
                    }
                  ],
                  "position": {
                    "start": { "line": 4, "column": 33, "offset": 74 },
                    "end": { "line": 4, "column": 39, "offset": 80 }
                  }
                },
                {
                  "type": "text",
                  "value": " an item!",
                  "position": {
                    "start": { "line": 4, "column": 39, "offset": 80 },
                    "end": { "line": 4, "column": 48, "offset": 89 }
                  }
                }
              ],
              "position": {
                "start": { "line": 4, "column": 3, "offset": 44 },
                "end": { "line": 4, "column": 48, "offset": 89 }
              }
            }
          ],
          "position": {
            "start": { "line": 4, "column": 1, "offset": 42 },
            "end": { "line": 4, "column": 48, "offset": 89 }
          }
        }
      ],
      "position": {
        "start": { "line": 2, "column": 1, "offset": 1 },
        "end": { "line": 4, "column": 48, "offset": 89 }
      }
    }
  ],
  "position": {
    "start": { "line": 1, "column": 1, "offset": 0 },
    "end": { "line": 5, "column": 1, "offset": 90 }
  }
}

Meanwhile, our AST from this text is:

Details

{
  "type": "root",
  "children": [
    {
      "type": "block",
      "children": [
        {
          "type": "list",
          "ordered": false,
          "spread": false,
          "children": [
            {
              "type": "listItem",
              "spread": true,
              "children": [
                {
                  "type": "text",
                  "value": "This is an item"
                }
              ]
            },
            {
              "type": "listItem",
              "spread": true,
              "children": [
                {
                  "type": "text",
                  "value": "This is also an item"
                }
              ]
            },
            {
              "type": "listItem",
              "spread": true,
              "children": [
                {
                  "type": "text",
                  "value": "Would you believe it, this is "
                },
                {
                  "type": "emphasis",
                  "children": [
                    {
                      "type": "text",
                      "value": "also"
                    }
                  ]
                },
                {
                  "type": "text",
                  "value": " an item!"
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

We're not embedding the Text inside a Paragraph.

Proposed solution

Follow the mdast spec, ensure we parse this properly

Additional notes

This is captured in jupyter-book/myst-spec#67 (comment), which I realised after opening the issue. We can fix it as part of the types changes.

N.b. I opened this issue when I noticed that tex-to-myst produces lists that don't export nicely to Markdown via --md.

[rowanc1]

In talking this through with @agoose77:

1. We don't process the spread boolean correctly (we don't use it at all in the renderer)
2. Depending on the list (tight or loose) the markdown parser sometimes adds a paragraph.
3. latex parsing does slightly different things to markdown parsing

The outcome we want:

• Always have a paragraph in a listItem
• Spread is processed correctly (with various test items)

This means updating the list items parsing in both markdown and latex. Possibly adding a transform (hopefully not necessary).