EntityParser can't handle encoded emoji

[aduth]

While the tokenizer will gracefully decode most encoded characters:

⇒ node
> var Tokenizer = require( 'simple-html-tokenizer' );
undefined
> Tokenizer.tokenize( '&' )[ 0 ].chars === '&'
true

It doesn't handle characters whose encodings exceed 16 bits (e.g. emoji):

⇒ node
> var Tokenizer = require( 'simple-html-tokenizer' );
undefined
> Tokenizer.tokenize( '😅' )[ 0 ].chars === '😅'
false

It may be that EntityParser should use String.fromCodePoint in place of String.fromCharCode instead, or an equivalent polyfill?

Related:

• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/fromCodePoint
• https://thekevinscott.com/emojis-in-javascript/

[rwjblue]

Seems reasonable to me...

@krisselden what do you think?

[krisselden]

Yes