lxml 5.0 Seems To Break preserve_handlebar_syntax

[CraigRobertWhite]

As we updated dependencies, lxml got updated to 5.0.0 which started replacing the spaces within the handlebars with %20. Our temporary fix was to pin lxml to 4.9.4 ourselves.

[Eboreg]

Thanks for pointing out that this has to do with the LXML version. A small correction, though: It seems that LXML encoded the spaces in prior versions too. The difference is that in 4.9.4 it encodes the handlebars as well, which it doesn't do in 5.0.

My source HTML contains this:

<a href="{{ redeemUrl }}" ...>...</a>

With LXML 5.0, my out variable contained this right before unescaping and unquoting (premailer.py:564):

<a href="{{%20redeemUrl%20}}" ...>...</a>

However, with LXML 4.9.4, the same part looked like this:

<a href="%7B%7B%20redeemUrl%20%7D%7D" ...>...</a>

And the re.sub() call on premailer.py:565 only looks for strings containing encoded handlebars ("%7B"), thus with LXML 5.0 it doesn't unescape the spaces.

I haven't dug any deeper into the reasons for this, but rather than downgrading LXML, I will awkwardly add this to my code right after running premailer.transform():

from html import unescape
from urllib.parse import unquote

# [...]

out = re.sub(
    r'="{{(.+?)}}"',
    lambda match: '="{{' + unescape(unquote(match.groups()[0])) + '}}"',
    out,
)