handle_ref() cannot handle &gid=

[ix5]

When you have links formatted like this: http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers=&discussionID=12741155&gid=87954&trk=EML_anet_qa_ttle-0Pt79xs2RVr6JBpnsJt7dBpSBA), the ?gid= part makes the pelican HTMLParser hiccup when truncating for feeds:
(This popped up while helping ramonsuarez on his bug #2258 )

CRITICAL: ValueError: substring not found
Traceback (most recent call last):
  File "/home/me/data/pip/lib/python3.6/site-packages/pelican/utils.py", line 556, in handle_entityref
    codepoint = html_entities.name2codepoint[name]
KeyError: 'gid'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/me/data/pip/bin/pelican", line 11, in <module>
    sys.exit(main())
  File "/home/me/data/pip/lib/python3.6/site-packages/pelican/__init__.py", line 487, in main
    pelican.run()
  File "/home/me/data/pip/lib/python3.6/site-packages/pelican/__init__.py", line 179, in run
    p.generate_output(writer)
  File "/home/me/data/pip/lib/python3.6/site-packages/pelican/generators.py", line 599, in generate_output
    self.generate_feeds(writer)
  File "/home/me/data/pip/lib/python3.6/site-packages/pelican/generators.py", line 300, in generate_feeds
    self.settings['FEED_ALL_ATOM'])
  File "/home/me/data/pip/lib/python3.6/site-packages/pelican/writers.py", line 123, in write_feed
    self._add_item_to_the_feed(feed, elements[i])
  File "/home/me/data/pip/lib/python3.6/site-packages/pelican/writers.py", line 52, in _add_item_to_the_feed
    description = item.summary
  File "/home/me/data/pip/lib/python3.6/site-packages/pelican/contents.py", line 310, in summary
    return self.get_summary(self.get_siteurl())
  File "/home/me/data/pip/lib/python3.6/site-packages/pelican/utils.py", line 173, in __call__
    value = self.func(*args)
  File "/home/me/data/pip/lib/python3.6/site-packages/pelican/contents.py", line 306, in get_summary
    self.settings['SUMMARY_MAX_LENGTH'])
  File "/home/me/data/pip/lib/python3.6/site-packages/pelican/utils.py", line 583, in truncate_html_words
    truncator.feed(s)
  File "/home/me/data/pip/lib/python3.6/site-packages/pelican/utils.py", line 484, in feed
    HTMLParser.feed(self, *args, **kwargs)
  File "/usr/lib/python3.6/html/parser.py", line 111, in feed
    self.goahead(0)
  File "/usr/lib/python3.6/html/parser.py", line 219, in goahead
    self.handle_entityref(name)
  File "/home/me/data/pip/lib/python3.6/site-packages/pelican/utils.py", line 558, in handle_entityref
    self.handle_ref('')
  File "/home/me/data/pip/lib/python3.6/site-packages/pelican/utils.py", line 543, in handle_ref
    ref_end = self.rawdata.index(';', offset) + 1
ValueError: substring not found

It looks like the issue was introduced in 9d0804de7: When truncating, consider hypens, apostrophes and HTML entities.

As I do not fully understand this, @andreacorbellini do you think this simple change to use find() instead of index() in handle_ref() is sufficient? I have been able to get pelican to work using this, but I don’t know whether this is the right approach.

And is &gid a protected codepoint somehow?

Maybe @mosra , sincce you're working on something related to unescaping right now, you could have a look at this?

[mosra]

Um, is the HTML source really looking like this?

<a href="http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers=&discussionID=12741155&gid=87954&trk=...">

Then that's an invalid HTML, as the &s should be escaped (even the GitHub syntax highlighter gets grumpy when seeing them, haha):

<a href="http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers=&amp;discussionID=12741155&amp;gid=87954&amp;trk=...">

I think what happens there is that the code is trying to find an HTML entity in an invalid markup and fails. So either:

• don't allow invalid markup and fail hard (like it's now)
• or be less optimistic and gracefully handle & characters that actually aren't HTML entities. I'm not sure what all the code does (I assume it's trying to not truncate in the middle of an entity?), but I think the ref_end should become offset + 1 when there's no matching ;. With ref_end = find(';', offset) + 1 it becomes 0, that's not correct.

[ix5]

Well, it's not HTML that is in ramon’s source files, it’s markdown but enclosed in <div> tags. So you might have

<div class="some-imported-posterous-crap" style"something......">
[linkedin.com](http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers=&gid=87954&discussionID=11957447&sik=1262886119376&trk=ug_qa_q&goback=%2Eand_87954_11957447_*2_*2%2Eana_87954_1262886119376_3_1)
</div>

I don’t know whether pelican should fail on that or not. Github does render the markdown correctly, see the repo here(the linkedin link at the end)

But yeah I agree with you, I do not understand the code as well, @andreacorbellini might have an idea.

[justinmayer]

Any thoughts on this topic, @andreacorbellini?

[andreacorbellini]

Hi, sorry for ignoring all these messages: I didn't realize I was being mentioned.

Here's an explanation of the code: when Python's HTML parser finds a sequence like —, it will call handle_entityref('mdash'), which in turn will call handle_ref('—'). The method handle_ref() will then attempt to find the index where — ends, in order to update the last_word_end attribute appropriately.

The way handle_ref() works right now that it tires to find the end of — by looking for the character ;. In this case, there's no such ;, hence it fails with ValueError. Whether this is correct HTML or not is not important because, as you have already noted, this is Markdown, not strict HTML.

The way I would fix handle_ref() is by making handle_entityref() and handle_charref() pass down the length of the entity ref being parsed. So, for example, when handle_entityref('mdash') is called, we call handle_ref('—', len('mdash')). Given this additional parameter, handle_ref() will be able to see whether there's a ; or not and calculate last_word_end accurately without throwing any exceptions.

[andreacorbellini]

PR submitted. Again, apologies for taking so long!

This will make the &stuff work in URLs, but I've noticed another (unrelated) problem with truncating Markdown: this string...

<div>
[linkedin.com](http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers=&gid=87954&discussionID=11957447&sik=1262886119376&trk=ug_qa_q&goback=%2Eand_87954_11957447_*2_*2%2Eana_87954_1262886119376_3_1)
</div>

... might get truncated like this:

<div>
[linkedin.com](http://www.linkedin.com/...
</div>

That is: the URL is truncated and not closed, even though it should not affect the word count. I guess we need a Markdown parser instead of an HTML parser?

[justinmayer]

Addressed by @andreacorbellini and merged. Many thanks! 🙏🏻