parley_engine: Always shape full text, ensuring ShapedText covers the full text

[tomcur]

To shape, you now call Shaper::shape_text, which owns the itemization loop. Together with font selection being infallible (#722), ShapedText is then guaranteed to represent the full source text contiguously. That's useful to simplify some bookkeeping. See #715 (comment) for some prior discussion on this. Because users previously looped items themselves and would call Shaper::shape_item, that meant items could be skipped or submitted out-of-order.

Because control flow moved into parley_engine, some structures are introduced. Shaper::shape_text takes an iterator of items encoding the span and holding the item's ShapeOptions (previously, parley constructed the options inside the loop and passed them as a parameter to Shaper::shape_item). Font selection is now a trait instead of a callback, with one method for selection and one method called at the start of each segment.

Note, "segment" is a new name, and encodes maximal spans within the user's items that have constant bidi level and script. The distinction between items and segments becomes more important later on, because browsers reset grapheme segmentation at item boundaries (where shaping options change), but not at segment boundaries.

Performance on the Japanese benchmarks is improved quite a bit, mostly because we're smarter about reusing the FontSelector. Performance on Arabic and Latin is about +1.5%, but addressing the TODO I left on the the item iterator in parley appears to make this a net small performance win on those scripts (i.e., iterating style runs and inline boxes to figure out where item boundaries are, instead of an O(chars) walk). That's for a separate PR: I just had an LLM address that TODO for an initial measurement, and have not reviewed that code at all.

Benchmarks with the state as in this PR

$ cargo bench --bench main -- compare ../target/benchmarks/main -t 8.

Default Style - arabic 20 characters               [   9.0 us ...   9.2 us ]      +1.42%*
Default Style - latin 20 characters                [   4.5 us ...   4.5 us ]      +1.62%*
Default Style - japanese 20 characters             [   8.6 us ...   8.0 us ]      -7.29%*
Default Style - arabic 1 paragraph                 [  48.6 us ...  49.1 us ]      +1.12%*
Default Style - latin 1 paragraph                  [  17.4 us ...  17.7 us ]      +1.75%*
Default Style - japanese 1 paragraph               [  72.6 us ...  64.9 us ]     -10.66%*
Default Style - arabic 4 paragraph                 [ 202.9 us ... 205.2 us ]      +1.11%*
Default Style - latin 4 paragraph                  [  65.7 us ...  66.3 us ]      +0.82%
Default Style - japanese 4 paragraph               [ 102.4 us ...  90.8 us ]     -11.28%*
Styled - arabic 20 characters                      [  10.1 us ...  10.2 us ]      +1.15%*
Styled - latin 20 characters                       [   5.6 us ...   5.7 us ]      +1.68%*
Styled - japanese 20 characters                    [   9.3 us ...   8.5 us ]      -8.89%*
Styled - arabic 1 paragraph                        [  51.0 us ...  51.7 us ]      +1.25%*
Styled - latin 1 paragraph                         [  21.9 us ...  22.3 us ]      +1.94%*
Styled - japanese 1 paragraph                      [  78.8 us ...  71.1 us ]      -9.77%*
Styled - arabic 4 paragraph                        [ 222.8 us ... 226.6 us ]      +1.72%*
Styled - latin 4 paragraph                         [  84.9 us ...  85.4 us ]      +0.57%
Styled - japanese 4 paragraph                      [ 111.6 us ... 101.0 us ]      -9.55%*

Experimental: benchmarks after addressing the item iterator TODO

$ cargo bench --bench main -- compare ../target/benchmarks/main -t 8.

Default Style - arabic 20 characters               [   9.0 us ...   8.9 us ]      -1.04%*
Default Style - latin 20 characters                [   4.5 us ...   4.5 us ]      -0.29%
Default Style - japanese 20 characters             [   8.7 us ...   7.8 us ]      -9.50%*
Default Style - arabic 1 paragraph                 [  48.6 us ...  48.3 us ]      -0.51%
Default Style - latin 1 paragraph                  [  17.5 us ...  17.5 us ]      +0.09%
Default Style - japanese 1 paragraph               [  72.5 us ...  63.3 us ]     -12.76%*
Default Style - arabic 4 paragraph                 [ 203.0 us ... 201.7 us ]      -0.64%
Default Style - latin 4 paragraph                  [  65.7 us ...  66.2 us ]      +0.64%
Default Style - japanese 4 paragraph               [ 102.5 us ...  88.8 us ]     -13.34%*
Styled - arabic 20 characters                      [  10.1 us ...  10.0 us ]      -0.78%
Styled - latin 20 characters                       [   5.7 us ...   5.6 us ]      -1.37%*
Styled - japanese 20 characters                    [   9.3 us ...   8.4 us ]      -9.73%*
Styled - arabic 1 paragraph                        [  51.3 us ...  50.7 us ]      -1.23%*
Styled - latin 1 paragraph                         [  21.9 us ...  21.9 us ]      +0.03%
Styled - japanese 1 paragraph                      [  78.9 us ...  69.4 us ]     -12.10%*
Styled - arabic 4 paragraph                        [ 222.8 us ... 220.8 us ]      -0.92%
Styled - latin 4 paragraph                         [  84.9 us ...  84.8 us ]      -0.15%
Styled - japanese 4 paragraph                      [ 111.7 us ...  97.8 us ]     -12.45%*

[taj-p]

LGTM!

[taj-p]

I struggled a bit with the new names in this file.

The flow today is roughly:

• callers pass Items to shape_text
• shape_text uses an Itemizer to split each item range into Segments
• each Segment produces one or more ShapedRuns

I found it surprising that an Itemizer produces Segments rather than Items. At the same time, Segmenter (as a rename) or similar doesn't quite feel right. Both Item and Segment also feel fairly generic 🤔

I'm wondering whether we should rename Item to ShapingContextSpan and Segment to ItemizedRun.

Then the flow would read:

• callers pass ShapingContextSpans to shape_text
• the Itemizer splits them into ItemizedRuns
• each ItemizedRun produces one or more ShapedRuns

I don't feel too strongly here - especially considering the TODO comments which suggest the naming and semantics may change significantly as this migration evolves. Please defer to your own best judgement here.

[tomcur]

Yeah, very fair. I've decided not to fix it here yet. It's likely the exact semantics are going to change a bit more.

Two bits of additional context perhaps help.

First, the distinction between the "item" and "segment" is not really a technical necessity, but it follows browser behavior: browsers reset grapheme segmentation at their "item" boundaries (style changes), but they don't reset on script changes (and Blink also doesn't reset on bidi changes). This moves us in that direction. But, if my understanding is correct, style changes don't technically need to reset grapheme segmentation, so it's just a choice.

Second, an item is user-provided, whereas segments are derived from the text.

At the same time, Segmenter (as a rename) or similar doesn't quite feel right.

I agree, especially because it may be confused with the icu4x segmenters. One option is to perhaps name everything "items" again, where users can provide additional item boundaries and perhaps can specify whether they want those to reset grapheme segmentation or not. ShapeOptions could be provided by some callback called by parley_engine with the item ranges (i.e., the user's boundaries, plus constant bidi and script).

[taj-p]

👍

[taj-p]

Do we plan to return an Error from shape_text in the future?

[tomcur]

Hm, perhaps we should.