[v10] Add character count countFunction option

[colinrotherham]

Description

This PR adds a new countFunction option to character counts to cater for server-side differences in:

• Line length differences due to \n versus \r\n
• Word counts that vary based on empty spaces and punctuations
• Trimming empty space before counting
• Native multi-byte string support

For example, services might already count multi-byte strings server-side (e.g. len() in Python) resulting in client-side count mismatches, yet upcoming support for grapheme countType is blocked by a 3rd party library integration

Support for a custom countFunction means teams can close this support gap

Checklist

• Tested against our testing policy (Resolution, Browser & Accessibility)
• Follows our coding standards and style guide
• CHANGELOG entry

[MatMoore]

Some thoughts on the interface here...

Currently there are two things that influence the choice of algorithm: the selection of countFunction, and whether segmenter is set or not. Wondering if it would be simpler to have explicit functions for graphemes as well as characters and words? In that case passing countType: "graphemes" would be syntactic sugar for passing countFunction: countFunctions.graphemes and the same goes for "characters" and "words".

It also seems slightly inconsistent that if you pass countType: "words" with a custom function, we pass through a segmenter that can do word segmentation, but if you don't pass the custom function, it doesn't use the segmenter.

Maybe if the interface is simplified to fn(text) the segmenter could be an internal implementation detail?

[colinrotherham]

Yeah, I partly did this because Intl.Segmenter also supports granularity: "word" 😮

So instead of countType: "words" we could keep maxwords but set granularity: "word"

I'll see what feedback comes from the GOV.UK Design System team

Maybe if the interface is simplified to fn(text) the segmenter could be an internal implementation detail?

Yeah, I'm glad you said. I reckon for now let's ditch the static property and keep it all internal, especially with a potential performance cost to creating a new segmenter on every keystroke (hence injecting it in)

[colinrotherham]

Feedback applied, thanks @MatMoore

Regarding word counts via the segmenter, I've messaged @romaricpascal and added a comment as the Unicode Default Word Boundary Specification varies quite significantly from /\S+/g

For example, consider this phrase:

My mother-in-law—Wait, what?

It matches only 3 words currently:

["My", "mother-in-law—Wait,", "what?"]

Yet using the segmenter with granularity: "word" it matches 6 words:

["My", "mother", "in", "law", "Wait", "what"]

[MatMoore]

Yeah, I'm glad you said. I reckon for now let's ditch the static property and keep it all internal, especially with a potential performance cost to creating a new segmenter on every keystroke (hence injecting it in)

Yeah I figured we wouldn't want to instantiate one each call. The way you've done it is what I was imagining.

That unicode word boundary spec seems like a cleaner way to do word counting if we were implementing from scratch, but I guess it's a potentially breaking change so maybe it's not worth it if we're keeping maxwords. Are we going to wait for the govuk implementation before we merge this?

[colinrotherham]

Thanks. Yeah it'll be discussed at their next dev catch up

What we know is that they've proposed for maxwords to be deprecated

We could take the countType option as an opt-in to use segmenter. But as this comment mentions that might mean upgrading /\S+/g to a better word boundary regex for older browsers that lack support

[colinrotherham]

They've not had chance to discuss anything character count related other than this comment:

• Add character count Intl.Segmenter support with customisable count function alphagov/govuk-frontend#6995 (review)

But that applies to #1899 primarily

For this PR all count functions now use the same interface so it's ready for another review

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
95.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[colinrotherham]

@anandamaryon1 I'll be adding a changelog entry into #1899 to wrap up this stack of work:

• [v10] Deprecate character count maxwords and add countType option #1893
• [v10] Add character count countType: "characters" option using Intl.Segmenter #1895
• [v10] Add character count countFunction option #1897
• [v10] Add character count countType: "words" option using Intl.Segmenter #1899

With this initial PR used to uplift some more recent GOV.UK Frontend changes:

• [v10] Refactor character count method to reduce repeated updates #1892

Plus this PR to allow components to configure functions within their options:

• [v10] Add support for functions in component configs #1896

---

We've contributed our improvements back to GOV.UK Frontend via:

• Add character count Intl.Segmenter support with customisable count function alphagov/govuk-frontend#6995