Add the space-discarding feature

[isuffix]

This adds the space-discarding feature as described at typst/typst#7350 (comment), although I have changed from "whether a writing system uses spaces between words" to "whether a writing system uses spaces at all."

I will leave this PR description short as the code itself contains a plenty of discussion of rationale and implementation considerations, along with my research into the usage of space characters in various writing systems.

There is a lot of writing here, so I would really appreciate help with checking for typos and inconsistencies, as it has become hard for me to consider everything with fresh eyes. I am very amenable to suggestions :)

I will also restate that I only speak English and while I have tried to do good research, I am not infallible. I would appreciate any input from native speakers of Chinese or Japanese or any of the other writing systems discussed in the PR.

I would also like to thank @r12a for his wonderfully detailed orthography descriptions and script comparison table, without which this PR would not be nearly as complete or authoritative. If you're reading this, I would love any feedback you could provide.

[r12a]

Tibetan comes to mind as an example of an orthography that doesn't use (ASCII) spaces but does have delimiters (syllable-based) which could break a line in the source code, but which should not incur an extra space when stitching things together (see Tibetan Orthography Notes). hth

[YDX-2147483647]

I've gone through the source code (except the emoji part). I believe the current algorithm is simple and robust enough. I only have some suggestions regarding doc comments and tests. See my individual comments.

More materials supporting the current algorithm

East Asian Width

UAX #14: Unicode Line Breaking Algorithm also uses East Asian Width to filter out East Asian characters.

The symbol $EastAsian stands for the set [\p{ea=F}\p{ea=W}\p{ea=H}] of characters with Fullwidth, Wide, or Halfwidth East Asian Width.

Pandoc

Pandoc's east_asian_line_breaks extension uses charWidth to determine if a soft break (e.g., single newline in markdown and typst) should be removed. However, we've argued in typst/typst#7350 (comment) that the rules for determining widths are too complicated and it's better to use East Asian Width directly.

LaTeX

LaTeX cannot be taken as a reference, because the implementations are limited by the technology. Specifically, the whitespace in 字\n“ should be discarded, but luatexja keeps it. And the whitespace in ”\nA should be kept as a word space, but xeCJK discards it. See typst/typst#792 (comment) for the tests.

Typst cjk-unbreak

As for typst packages, cjk-unbreak uses the following regex to determine if a character is CJ (Chinese + Japanese) and discards the space iff either side matches the regex.

[\p{Han}，。；：！？‘’“”（）「」【】…—\p{Hiragana}\p{Katakana}]

This package is designed only for CJ, so the regex includes a few characters that are considered YesOrAmbiguous in this PR.
The algorithm in this PR is designed for all writing systems, so the difference to cjk-unbreak is acceptable.

Typst cjk-spacer

A newer typst package, cjk-spacer, uses a more complex algorithm. If I understand correctly, then its algorithm is equivalent to the following.

#let default-cjk-regex = regex(
  "["
    + "\p{scx:Hira}\p{scx:Kana}\p{scx:Han}\p{scx:Hang}\p{scx:Bopo}"
    + "\u3000-\u303F" // CJK Symbols and Punctuation
    + "\u3190-\u319F" // Kanbun
    + "\u31C0-\u31EF" // CJK Strokes
    + "\u3200-\u32FF" // Enclosed CJK Letters and Months
    + "\u3300-\u33FF" // CJK Compatibility
    + "\uFE10-\uFE1F" // Vertical Forms
    + "\uFE30-\uFE4F" // CJK Compatibility Forms
    + "\uFE50-\uFE6F" // Small Form Variants
    + "\uFF00-\uFFEF" // Halfwidth and Fullwidth Forms
    + "]",
)
#let default-western-open-punc-regex = regex(
  "[\p{Pi}\p{Ps}--["
    + "\u3000-\u303F" // CJK Symbols and Punctuation
    + "\uFE10-\uFE1F" // Vertical Forms
    + "\uFE30-\uFE4F" // CJK Compatibility Forms
    + "\uFE50-\uFE6F" // Small Form Variants
    + "\uFF00-\uFFEF" // Halfwidth and Fullwidth Forms
    + "]]",
)
#let default-western-close-punc-regex = regex(
  "[\p{Pf}\p{Pe}\p{Term}--["
    + "\u3000-\u303F" // CJK Symbols and Punctuation
    + "\uFE10-\uFE1F" // Vertical Forms
    + "\uFE30-\uFE4F" // CJK Compatibility Forms
    + "\uFE50-\uFE6F" // Small Form Variants
    + "\uFF00-\uFFEF" // Halfwidth and Fullwidth Forms
    + "]]",
)

#let discard_space_between(before, after) = {
  if after.matches(western-open-punc-regex).len() == 0 and after.starts-with(cjk-regex) {
    true
  } else if before.matches(western-close-punc-regex).len() == 0 and before.ends-with(cjk-regex) {
    true
  } else {
    false
  }
}

The cjk-spacer algorithm does not merely consider the characters immediately adjacent to the space, but rather the text segments around the space. This approach is appropriate when typesetting a document, but it's too surprising at the syntax level.

Also, cjk-spacer treats K the same as CJ. According to previous feedbacks in typst/typst#7350, discarding spaces is not preferable for Korean texts.

And I haven't check if the Unicode blocks enumerated by cjk-spacer are equivalent to this PR, but I think it's worth checking before merging this PR.

[YDX-2147483647]

The term ideographic punctuation needs clarification.

• There's a Unicode block called Ideographic Symbols and Punctuation, but obviously that isn't what you mean.

• ‼ is listed in both Punctuation marks in Chinese in CLReq and Character Classes in JLReq, but it isn't considered as a ideographic punctuation.

It looks like that you use this term colloquially. I suggest putting the relevant test cases in a separate function and linking to it. (similar to test_spacing_emoji_presentation)

[isuffix]

I'll split into two test functions and integrate the "Miscellaneous" section into the two punctuation groups.

Do you think we should consider special-casing ⁇ or ‼?

[YDX-2147483647]

Just a comment for your information: I get the narrow ¥ when I type in Chinese mode on my mobile phone, but I get the fullwidth ￥ when I press Shift+4 ($) in Chinese mode on my desktop computer.

[laurmaedje]

Logic-wise, this seems reasonable and very well supported in terms of rationale. I'd trust in the research and the other involved people's judgement. I just have some minor nits.

[isuffix]

I've updated comments and tests based on all of the feedback!

I think for this initial PR we will not include Tibetan. But one of the main points for moving this to Codex is so we can add GitHub issues and iterate at a separate pace than typst/typst. So we should add an issue for Tibetan support and discuss it more once this is merged.

For cjk-unbreak, it looks there are quite a few differences to this PR. The characters it doesn't include are all either Yi only or are ScriptExtensions=Common that have EastAsianWidth=Wide. For the other way around, this UnicodeSet link grouped by EastAsianWidth with Script=Hangul removed, shows 89 characters.

Of those, it seems the groups we should consider are:

1. The fullwidth latin letters: U+FF21-U+FF5A (Ａ - ｚ)
2. The circled numbers on black squares: U+3248-U+324F (㉈-㉏)
3. The six CJ marks with Sc=Inherited and EA=W: U+302A-U+302D and U+3099-U+309A
4. The corner tone marks: U+A700-U+A707 (Wikipedia)
5. U+303F IDEOGRAPHIC HALF FILL SPACE

I'll prepare an update to include these.

UnicodeSet syntax for this PR and cjk-unbreak

This PR currently:

[
  \p{Han} \p{Kana} \p{Hira} \p{Bopo} \p{Yi}
  [ \p{Common} & [\p{EA=H}\p{EA=F}\p{EA=W}] - \p{Emoji} - [₩] ]
]

cjk-unbreak default (excluding \p{Scx=Hang}):

[
  \p{Scx=Hira} \p{Scx=Kana} \p{Scx=Han} \p{Scx=Bopo}
  [ \u3000-\u303F \u3190-\u319F \u31C0-\u31EF \u3200-\u32FF \u3300-\u33FF \uFE10-\uFE1F \uFE30-\uFE4F \uFE50-\uFE6F \uFF00-\uFFEF ]
]

[isuffix]

I've looked into these character groups some more and I've updated the implementation based on them. Here are my changes for each set and some of my thoughts about their East Asian Width (EA) and Script Extension (Scx) properties along with the Unicode standard's description for some of them.

1. Fullwidth latin letters

I added the fullwidth latin letters. It feels odd that they don't have Scx=Han (or something) to indicate they may be used in a CJK context outside of which block they're a part of.

2. Circled numbers on black squares

I added the circled numbers on black squares. I have no clue why these would be EA=Ambiguous instead of Wide. I cannot imagine why a font would display a square containing two digits as a narrow character.

Here is Unicode's description of these characters (section 22.10.2, paragraph 2):

The enclosed symbols in the range U+3248..U+324F, which consist of circled numbers ten through eighty on white circles centered on black squares, are encoded for compatibility with the Japanese television standard, ARIB STD B24. In that standard, they are intended to represent symbols for speed limit signs, expressed in kilometers per hour.

3. The Six CJ-specific combining marks

I added the six combining marks. These only slipped by the initial definition because they're in Script=Inherited instead of Script=Common. I checked, and all other Inherited codepoints are EA=Ambiguous or EA=Neutral.

It may be worth using language information to make determinations on other Script=Inherited characters in the future (mostly combining marks and variation selectors), or including them in grapheme cluster analysis, but that will require more research.

4. Chinese corner tone marks

I'm ignoring the corner tone marks since they're not for Chinese typesetting, but for transcription of Chinese into English, which is why their Scx is Han|Latin. They also seem to be largely historical. Here's a document from 1841 which uses them.

Here is Unicode's description of these marks (section 7.8.2, paragraph 2):

The characters in the range U+A700..U+A707 are corner tone marks used in the transcription of Chinese. They were invented by Bridgman and Wells Williams in the 1830s. They have little current use, but are seen in a number of old Chinese sources.

5. 〿 U+303F IDEOGRAPHIC HALF FILL SPACE

(Note that the character is not failing to render, it is supposed to look like a vertical box with an X through it.)

I'm not adding this character as it doesn't actually seem to be used in Chinese or Japanese writing.

Conflicting Properties

This character gave me a lot of trouble because it has two seemingly conflicting property values in Unicode: Script_Extensions=Han and East_Asian_Width=Neutral. In UAX #11, the definition of Neutral is:

Neutral (Not East Asian): All other characters. Neutral characters do not occur in legacy East Asian character sets. By extension, they also do not tend to occur in East Asian typography.

It makes no sense for a character to have "no basis in East Asian typography" but also be used in the Han script!

It's also very confusing that 〿 U+303F is adjacent to 〾 U+303E IDEOGRAPHIC VARIATION INDICATOR, which is a wholly unrelated character used in ideographic description sequences. Trying to look for info about 〿 U+303F down that route is a dead end. But to make it worse, Unicode gives these two a custom subheader in their block as Special CJK Indicators despite having no historical similarity!

Judging by its name alone you would think, "oh, this is used in ideographic contexts, of course it should have Scx=Han, so EA=Neutral is the issue." But I think that IDEOGRAPHIC in its name is actually a misnomer. Let me explain.

Origins

The ideographic half fill space is a very old character in Unicode. It was present even in Unicode 1.0, and while I could not find any records documenting its choice of position in the CJK Symbols and Punctuation block, I did find that its real name used to be "Box X (DBCS Fill Character)" and its real codepoint was 000B when it lived in the IBM code pages for various CJK DOS systems using Double Byte Character Sets.

You can find this from the character's GCGID of SP500000 on this page from ccsids.net. And the character is visible in several code pages on Wikipedia such as page 1040 or page 1043 or the first two rows of page 897. Although I only got to those places by first finding it in the legacy external mapping tables (p121) for Unicode 1.0.

When you consider that the DOS code pages were originally designed for terminal-based systems using monospace fonts where characters would render in either one vertical cell or two (based on using either one or two bytes), Unicode's description of the ideographic half fill space starts to make more sense (section 6.2.13, paragraph 3):

U+303F IDEOGRAPHIC HALF FILL SPACE is a visible indicator of a display cell filler used when ideographic characters have been split during display on systems using a double-byte character encoding. It is included in the Unicode Standard for compatibility.

(This description has been present since Unicode 2.0.0 and has not changed)

The character is literally just an X in a box for filling empty cells! It's only "ideographic" because it was only used in code pages for CJK systems, but in that usage it was really a block element character similar to █ U+2588 FULL BLOCK or ▓ U+2593 DARK SHADE (the latter usually sharing an IBM code page with 〿 U+303F at position 0014).

Resolution

So of the two conflicting properties, only EA=Neutral makes sense! The character does render as narrow, and it doesn't really have a use in East Asian typography in the modern day. It's really Scx=Han that's incorrect. And if you look at the data files for Script_Extensions, 〿 U+303F is given Scx=Han as part of a range with 〾 U+303E, so they seemingly weren't considered separately.

And if you really go looking, you can find reference to it in revision 3 of UTR #11 (June 1999), where Asmus Freytag notes that 〿 U+303F is an exception for an old paragraph in the Unicode standard, as it is a non-full-width character in the set of given blocks, and he updates its East_Asian_Width from Wide to Neutral! But no explanation of its exceptionality is given, and the Background section was removed from later revisions of UTR #11 (which eventually became UAX #11). The closest I could find to a mention of it is in the meeting minutes for UTC #78 (p10) (Dec 1998) with some discussion of accepting the—at-the-time draft—UTR as a full technical report. I also found an email with editorial comments by John Cowan from that era (June 1998), but no mention of 〿 U+303F. Freytag's original draft paper for the 'Width' property may also be of interest.

So I might submit a proposal to have Scx=Han removed from U+303F and to note somewhere its similarity to the block element characters.

Won sign

I also found some more context on the won sign in UAX #11:

Note: Unlike U+00A5 ¥ YEN SIGN, U+20A9 ₩ WON SIGN has an explicit East_Asian_Width property value of East Asian Halfwidth (H). What makes U+00A5 different is that this character was included in a very common—and non–East Asian—character set standard, specifically ISO/IEC 8859-1, and encoded at 0xA5. Almost all legacy Latin fonts supported ISO/IEC 8859-1 in its entirety, using variable-width glyphs. By contrast, most legacy font implementations used an explicit half-width glyph for the won sign, whose source is the standard KS X 1003, and encoded at 0x5C. The assignment of the East Asian Halfwidth (H) property value does not preclude font developers from using a variable-width glyph for U+20A9, and doing so has become a common practice.

This distinction was also discussed many years prior in the meeting minutes for UTC #78 (p10), where the answer was: "Because ISO8859-1 has Yen sign. No other character set has legacy half-width Won sign."

To be honest, it actually makes me a little mad that the won sign isn't Ambiguous, and I may submit a proposal to change it. U+20A9 ₩ is the only character with EA=Halfwidth whose name doesn't start with "HALFWIDTH" as part of the Halfwidth And Fullwidth Forms block. I understand that it does have an explicit fullwidth form like the other halfwidth characters, but if its exceptionality has to be called out while other Ambiguous characters already need to be specialized by implementations, then what good is it actually doing with its Halfwidth designation?

[YDX-2147483647]

1. Fullwidth latin letters

I added the fullwidth latin letters. It feels odd that they don't have Scx=Han (or something) to indicate they may be used in a CJK context outside of which block they're a part of.

I think fullwidth Latin letters should be YesOrAmbiguous.

• Indeed, they are usually used in CJK context. However, even in CJK context, they are still used to write Latin languages. In other words, spaces between fullwidth Latin letters should be kept.

• Some OCR softwares do not distinguish between narrow and fullwidth Latin letters when a Chinese article contains a few English phrases and sentences. The software might output fullwidth characters even if the letters look narrow. (One may still enable the pwid OpenType feature to display the fullwidth letters as narrow.) In that case, spaces between them should be kept.

[isuffix]

Thank you, I was not aware of that context. I've reverted the fullwidth letters.