[help] Straight forward standardized shorthand to decode HTML entities

[forthrin]

Please show your code!

CGI.unescapeHTML('Røde')
"Røde" # Fails

HTMLEntities.new.decode('Røde')
"Røde" # Works, but requires an extra library not used otherwise

Nokogiri.HTML5('Røde').text
"Røde" # Works, but must go via full DOM document

Nokolexbor.HTML('Røde').at('html').text
"Røde" # Works, but must go via full DOM document (and look up html/body tag, a difference from Nokogiri)

What would be an ideal official convenience shorthand in Nokogiri, save everyone making their own refinement?

Environment

ruby 2025-08-01 30a20bc16
rubocop 2025-09-06 402f0cfcf
System Version: macOS 15.6 (24G84)
Model Identifier: MacBookAir10,1

[stevecheckoway]

Hi @forthrin,

Nokogiri's HTML5 parser does have a function which is designed to identify HTML entities (or character references as the HTML standard calls them) and returns what they map to but on its own, that's not sufficient to get what you want. Some of the tricky bits of doing this that come to mind

• Entities are handled differently depending on if they're in an attribute or not;
• Entities can be erroneous and not expand into anything other than themselves (e.g., &#z) others can be erroneous and still expand to the expected text (e.g., &#x80 (missing a semicolon) expands to U+20AC (€));
• Entities can be syntactically correct but still lead to errors like �

Nokogiri's HTML5 parser deals with this by implementing the HTML tokenization which handles all of that. Adding what you're asking here would amount to reimplementing a portion of the tokenizer specifically to handle entities.

As an aside, the htmlentities gem doesn't handle all of the HTML5 entities (nor is it attempting to). E.g., =⃥ should expand to U+0003D U+020E5 (=⃥).

[forthrin]

This came up again, so did a test on all entities found in a cache of HTML files, and all except CGI do well on typical specimens.

Possible actions:

• CGI: Support a few more of the most frequent entities
• Nokos: Don't convert misnomers such as &degree; (?)
• Nokolexbor: Don't require DOM for conversion

@stevecheckoway: Awaiting your comments.

Freq	Entity	Point	CGI	HTMLEntities	Nokogiri	Nokolexbor
648019	&	U+0026	&	&	&	&
417737	"	U+0022	"	"	"	"
13070	&lt;	U+003C	<	<	<	<
13061	&gt;	U+003E	>	>	>	>
1030	&apos;	U+0027	'	'	'	'
176311	&nbsp;	U+00A0	&nbsp;	U+00A0	U+00A0	U+00A0
20192	&copy;	U+00A9	&copy;	©	©	©
143	&raquo;	U+00BB	&raquo;	»	»	»
132	&laquo;	U+00AB	&laquo;	«	«	«
54	&deg;	U+00B0	&deg;	°	°	°
3	&reg;	U+00AE	&reg;	®	®	®
4336	&aring;	U+00E5	&aring;	å	å	å
3659	&oslash;	U+00F8	&oslash;	ø	ø	ø
173	&aelig;	U+00E6	&aelig;	æ	æ	æ
67	&eacute;	U+00E9	&eacute;	é	é	é
5144	&rsquo;	U+2019	&rsquo;	’	’	’
777	&ndash;	U+2013	&ndash;	–	–	–
743	&bull;	U+2022	&bull;	•	•	•
222	&hellip;	U+2026	&hellip;	…	…	…
113	&rsaquo;	U+203A	&rsaquo;	›	›	›
113	&lsaquo;	U+2039	&lsaquo;	‹	‹	‹
37	&mdash;	U+2014	&mdash;	—	—	—
14	&dagger;	U+2020	&dagger;	†	†	†
1	&permil;	U+2030	&permil;	‰	‰	‰
192	&euro;	U+20AC	&euro;	€	€	€
3	&trade;	U+2122	&trade;	™	™	™

[stevecheckoway]

Hi @forthrin,

I'm not sure I understand exactly what you're asking me to do. Writing a wrapper function to call Nokogiri appropriately and extract the result seems like a simple 2-line function. I'll defer to Mike on whether that's something he wants to include in Nokogiri or not but my inclination would be not to. (It could be done in another gem as well, but that seems like a left-pad situation so probably not a great idea either.)

If the problem is Nokogiri builds a full DOM, then well, yeah. That's what it does.

If you wanted a function that took a single HTML entity and decoded it, I could see exposing that functionality from Nokogiri (that would be the internal function match_named_char_ref()) but I don't think that's what you want based on your examples.

You want to take a restricted set of HTML and then decode it to text. But I don't know what that restricted set is or what behavior your intended decoding function should have in the presence of other HTML markup. E.g., what would you like <span>&amp;</span> to return? What about broken markup? What about 0-bytes? What about other errors?

[nwellnhof]

Nokogiri.HTML5('Røde').text

The problem with this approach is that it will consume leading white-space. You have to replace < with < and prepend a <body> tag. Then it should be parsed as a single text node as you would expect.

[stevecheckoway]

Nokogiri.HTML5('Røde').text

The problem with this approach is that it will consume leading white-space. You have to replace < with < and prepend a <body> tag. Then it should be parsed as a single text node as you would expect.

require 'nokogiri'

def decode(input)
  return Nokogiri::HTML5::DocumentFragment.new(Nokogiri::HTML5::Document.new,
                                               input,
                                               context:'textarea')
    .text
end

while line = gets
  puts(decode(line.chomp))
end

This essentially 1-line function seems to work for me. Caching and reusing the Document might be worthwhile, but I'd definitely test that.

[forthrin]

OK, I suddenly realize the need to go into the relevant use cases, because normally you get this for free with a regular HTML document, so when do you need to decode explicitly apart from that?

• Podcast descriptions with HTML and entities
• HTML so last century that all efforts of using CSS paths crash and burn, reverting to RegExp and ultimately unescaping.
• Sites who think it's a good idea to pass HTML snippets in XHR JSON values rather than pure data.

And these are usually always the same few culprit entities, thus the request.

[stevecheckoway]

OK, I suddenly realize the need to go into the relevant use cases, because normally you get this for free with a regular HTML document, so when do you need to decode explicitly apart from that?

* Podcast descriptions with HTML and entities

* HTML so last century that all efforts of using CSS paths crash and burn, reverting to RegExp and ultimately unescaping.

* Sites who think it's a good idea to pass HTML snippets in XHR JSON values rather than pure data.

And these are usually always the same few culprit entities, thus the request.

Hi @forthrin,

You didn't respond to these questions which makes it difficult to help:

You want to take a restricted set of HTML and then decode it to text. But I don't know what that restricted set is or what behavior your intended decoding function should have in the presence of other HTML markup. E.g., what would you like & to return? What about broken markup? What about 0-bytes? What about other errors?

[forthrin]

The restricted set I've outlined in OP, ie. quarterblocks 0080, 2000 plus the couple of currencies. Latin 1 would also be a good bonus because of it's legacy prevalence.

Whenever I've encountered broken HTML, I've usually just gotten rid of all of it with a quick gsub, because it has never had any value in itself, gsub(%r{</?.*?>}, '').

A single & would remain a single &. 0-bytes I've never encountered in the wild either. I supposed they would remain as-is. Any particular reason you bring these up, except covering your bases for every possible scenartio?

[stevecheckoway]

So if I understand correctly, you're happy removing all other HTML markup leaving just plain text and entities.

I mentioned the broken HTML and 0 bytes because the standard gives meaning to nearly every sequence of bytes, it's just that many of these result in errors.

Given that, I'd recommend the decode function I gave above, potentially with your gsub call to remove all start and end tags.

If that's too slow for your use case, then I think the options are

1. Improve the parsing performance of the gumbo library; or
2. Implement a stand-alone portion of the tokenizer.

For approach 1, I'd suggest looking at all of the places where adding character just appends it to a string buffer and doesn't change state and handle that in a loop rather than going through the state dispatch mechanism (which is surprisingly complicated).

For approach 2, I'd recommend implementing the RCData state other than the handling for <.

Note that this approach is what my decode function is approximating. Specifically, it's parsing a fragment in the context of a textarea HTML element. The rules for that say to start the tokenizer in the RCData state and it'll stay in that state or the character reference states until the end of the input. The title element behaves similarly and could be used instead of textarea.

[forthrin]

Both Nokogiri and Nokolexbor already support all entities known to man:

curl -s https://html.spec.whatwg.org/entities.json | jq -r '. | keys[]' | tr -d '\n' | ruby -r nokogiri -e 'puts Nokogiri.HTML5(STDIN).text'

So this seems to come down to two things.

1. Performance: How much overhead adds Nokogiri.HTML5().text and Nokolexbor.HTML(').at('html').text?
2. Terseness: What options are there for convenience functions? Are refinements OoS for libraries and left to each dev?

module Nice
  refine String do
    def dehtml
      Nokogiri.HTML5(self).text # gsub(%r{</?.*?>}, '')
    end
  end
end

puts '<blink>R&oslash;de</blink>'.dehtml # Røde