Map prepended text to a zero-width span to avoid overlapping offsets (#1775)

[v-code01]

Problem

char_to_token() can point to the wrong token for tokenizers that use a Metaspace-style prefix (e.g. DebertaV2TokenizerFast, XLMRobertaTokenizerFast), because token offsets overlap on multi-byte scripts. Reported in #1775.

Root cause is in NormalizedString::prepend. It mapped the prepended text — which has no counterpart in the original input — onto the first original character's byte range. That is invisible while the prefix stays glued to the following text, but once the model splits the prepended marker into its own token it surfaces. For "中文" under Metaspace (xlm-roberta), the tokens are ["▁", "中文"] and the offsets were:

▁    (0, 3)   ← borrows 中's bytes
中文  (0, 6)   ← overlaps

so char_to_token(0) resolves to ▁ instead of 中文. Reproduces on Chinese and Japanese samples; Latin/Cyrillic/Korean don't split the ▁word unit, so they never exposed it.

Fix

Prepended text has no source in the original string, so it should map to a zero-width span at the start. prepend now inserts the text as new characters at an empty range, leaving the existing characters' offsets untouched:

▁    (0, 0)   ← zero-width, synthetic
中文  (0, 6)   ← no overlap

ByteLevel's byte-level remap is updated to transform over the full normalized range instead of the original range, so the now-zero-width prefix byte (e.g. the prefix space from add_prefix_space) is still covered — otherwise it would be excluded and desync the transformation offsets.

Tests

• New prepend_multibyte_no_overlap regression test (the exact CJK + ▁ case).
• Updated the prepend and Prepend-normalizer alignment expectations to the corrected zero-width behaviour.
• Full suite green (cargo test, with make test data), including all byte_level, added_tokens, and normalizer tests. cargo fmt / cargo clippy clean.

Every prepend caller in the tree (Metaspace, ByteLevel's leading space, the Prepend normalizer) prepends synthetic text, so zero-width alignment is correct for all of them.

[v-code01]

cc @ArthurZucker @McPatate — this fixes #1775 (overlapping offsets → char_to_token pointing at the wrong token) at its root in NormalizedString::prepend: prepended text mapped onto the first original character's bytes, which surfaced as an overlap once the model split the metaspace ▁ into its own token on CJK. Prepend now maps synthetic prefixes to a zero-width span; ByteLevel's byte remap was adjusted to the full normalized range so the prefix space is still covered. Full suite + make test data green, fmt/clippy clean.