Multi-line shortcodes end lists too early

[cpg314]

Bug Report

I think this is identical to #1355, but given that issue is 3 years old, I am reopening an issue at @Keats 's request.

In summary, shortcodes that span multiple lines will result in lists closing too early and generating invalid HTML.

Environment

Zola version: 0.19.2 (1aa21e0)

Step to reproduce

content/_index.md

+++
template = "index.html"
+++

Test

- A
  {{image()}}
- B
- C

template/index.html

{{ section.content | safe }}

template/shortcodes/image.md

<div>
</div>

Expected Behavior

If the shortcode is put on a single line, then the generated code is correct:

<p>Test</p>
<ul>
<li>
<p>A</p>
<div> </div>
</li>
<li>
<p>B</p>
</li>
<li>
<p>C</p>
</li>
</ul>

Current Behavior

The generated code is:

<p>Test</p>
<ul>
<li>A<div>
</li>
</ul>
</div>
<ul>
<li>B</li>
<li>C</li>
</ul>

Note the </ul> closing too early.

[ARitz-Cracker]

Notes for myself (and anyone else) in case this issue is looked more in-depth.

Unsure If this is underlying behavior of the pulldown-cmark dependency

Markdown:

- A
  a
- B
- C

Generated HTML:

<ul>
<li>A a</li>
<li>B</li>
<li>C</li>
</ul>

Unsure if this is expected, github, for example, puts a <br>.

Markdown:

- A
  
  a
- B
- C

Generated HTML:

<ul>
<li>
<p>A</p>
<p>a</p>
</li>
<li>
<p>B</p>
</li>
<li>
<p>C</p>
</li>
</ul>

Works as expected, the double-space starts a new paragraph in the list item.

Markdown:

- A
  
  {{image()}}
- B
- C

Generated HTML:

<ul>
<li>
<p>A</p>
<p><div></div></p>
</li>
<li>
<p>B</p>
</li>
<li>
<p>C</p>
</li>
</ul>

Works as expected, the div is inside the paragraph, which is inside the list item.

Therefore, a potential source of the issue is how multi-line list items are generally handled outside of paragraphs.

[dawoonchung]

I dug up on this a bit, and it seems that the issue is coming from the way pest parser handles new line and indents. Markdown list requires list element to be in the same indent level, e.g.:

- item 1
- item 2
  <div>Custom HTML item</div>
^^ indent required
- item 3

But it seems that when markdown shortcodes are parsed through pest parser, the indentation is trimmed along with the new line (\n).

I didn't have time to look into pest parser or to come up with a solution, but leaving this comment here in case someone wants to pick this up.

[kshitijaucharmal]

Hey, I would like to work on this issue, could you please assign it to me?
Also, thanks @dawoonchung for the insightful comment.

[Keats]

Done

[kshitijaucharmal]

Digging around in the code, I found this file components/utils/src/templates.rs having the get_shortcodes() function.
the template variable in the for loop in this function contains the string version of the shortcode, and so I am trying to modify it by removing \n from between tags.

I think this is not the way to go, as this function should not handle the shortcode formatting. Can anyone point me in the direction I need to go to solve this? Thanks :)

[thefiredman]

Digging around in the code, I found this file components/utils/src/templates.rs having the get_shortcodes() function. the template variable in the for loop in this function contains the string version of the shortcode, and so I am trying to modify it by removing \n from between tags.

I think this is not the way to go, as this function should not handle the shortcode formatting. Can anyone point me in the direction I need to go to solve this? Thanks :)

Hey, any progress on this? Thinking of diving in myself.

Not sure if this is a separate issue, however the issue also seems to occur when a shortcode (itself) indents more than three layers deep with any elements: (solution don't indent)

<div>
   <div>
       <div>
          {{ body | markdown | safe }}
        </div>
    </div>
</div>

The current behavior is broken.
Looking at it again, I might guess it is a deeper problem then the newlines.
Looking at it again, it's because of the markdown filter lol.

[Keats]

If only it was possible to disable the indented code blocks, that would make things simpler ;(

[thefiredman]

If only it was possible to disable the indented code blocks, that would make things simpler ;(

Seems weird, wouldn't ya filter out the whitespaces before parsing it. Prolly in ignorance land rn.

[ISSOtm]

Indentation and newlines are important, so they cannot be stripped.

IMO, the fix would be to indent each of the shortcode's lines by as much as the first line, and trim as much leading indentation from the body's.

This may appear as insufficient in some cases:

- A
  {% ok() %}
  B
  {% end %}
- {% not_ok() %}
  C
  {% end %}

...but the spec allows list items to start with a blank line, so this would work:

-
  {% now_it_s_ok() %}
  C
  {% end %}

If no objections are raised, I can look into implementing this—it's breaking a page of my own at this time :P

---

Unsure if this is expected, github, for example, puts a <br>.

@ARitz-Cracker It's normally not rendered (https://spec.commonmark.org/0.31.2/#soft-line-breaks), but A renderer may also provide an option to render soft line breaks as hard line breaks. so that's what GitHub is doing in discussions (try it in a README.md or a wiki!)

[Keats]

What would the fix look like?