Replacement in single-byte encodings should not produce two output bytes per one input codepoint

[ChALkeR]

> [...String.fromCodePoint(0x10000)].length
1 // 1 codepoint
> require('iconv-lite').encode(String.fromCodePoint(0x10000), 'utf32', { addBOM: false })
<Buffer 00 00 01 00> // 1 codepoint
> require('iconv-lite').encode(String.fromCodePoint(0x10000), 'windows1252').toString()
'??' // two replacements?

This happens for all codepoints above 0xFFFF

Single-byte encodings map each byte to a codepoint directly, there is no reason for the encoder to spawn two replacements there

[vip-delete]

there is no reason for the encoder to spawn two replacements

speed is the only reason

I’m afraid, this makes iconv-lite much slower, since it needs to support leftovers for SBCS, such as "\uD83D\uDE0A".

encoder.write("\uD83D");
encoder.write("\uDE0A");
encoder.end();

This should produce [0x3F] only.

[ChALkeR]

@vip-delete Performance is not a reason to break validity

And that argument is not even true - @exodus/bytes is both significantly faster and correct for other cases
Quick bench: https://gist.github.com/ChALkeR/c80ddb27b226bc9072e9ed3e858be70f

Note: replacements in encoders are not yet shipped there but use the same codepath.
Valid data would still be as fast in replacement mode, there is no perf degradation from that.

Datasets are from https://github.com/lemire/unicode_lipsum

[ChALkeR]

Quick test on replacement mode in draft:

single-byte replacement iso-8859-9 (emoji): @exodus/bytes x 19,349 ops/sec @ 51μs/op (46μs..621μs)
single-byte replacement iso-8859-9 (emoji): iconv-lite x 14,010 ops/sec @ 71μs/op (69μs..836μs)

This is on https://github.com/lemire/unicode_lipsum/blob/main/lipsum/Emoji-Lipsum.utf8.txt
Produced lengths: @exodus/bytes: 16384, iconv: 32768, [...string].length: 16384

[vip-delete]

@ChALkeR can you share "single-byte replacement iso-8859-9 (emoji)" test please?
@exodus/bytes doesn’t even have a replacement mode, does it?

import { createSinglebyteEncoder } from "@exodus/bytes/single-byte.js";
const encoder = createSinglebyteEncoder("iso-8859-9");
const buf = encoder("😊");
console.log(buf); // ERROR: Input is not well-formed for this encoding

[ChALkeR]

@vip-delete as I said in #368 (comment), replacement mode for encoders is not merged yet, its a draft

I’ll file it as a PR later today and will post a link here

[ChALkeR]

@vip-delete here it is: ExodusOSS/bytes#18
It's still a draft there and needs test/cleanup/improvements/optimiziations before it can land

Non-Node.js codepath is not optimized atm