[v10] Add character count countType: "words" option using Intl.Segmenter

[colinrotherham]

Description

This PR extends #1895 to add character count support for countType: "words" using Intl.Segmenter

  {{ characterCount({
    label: {
      text: "Can you provide more detail?"
    },
    name: "more-detail",
-   maxwords: 200
+   maxlength: 200,
+   countType: "words"
  }) }}

Issues with separator regex

The existing word count behaviour uses /\S+/g to count all consecutive non-whitespace characters

But consider this phrase where hyphens and em-dashes are used as separators:

My mother-in-law—Wait, what?

It matches only 3 words currently:

["My", "mother-in-law—Wait,", "what?"]

Yet using the segmenter with granularity: "word" it matches 6 words:

["My", "mother", "in", "law", "Wait", "what"]

To solve this, but avoid breaking changes:

• Setting countType: "words will use Intl.Segmenter in modern browsers
• Setting maxwords (deprecated) will use the existing word count behaviour

But what about older browsers that lack Intl.Segmenter support?

I've updated the word separator regex based on this comment: alphagov/govuk-frontend#1364 (comment)

Checklist

• Tested against our testing policy (Resolution, Browser & Accessibility)
• Follows our coding standards and style guide
• CHANGELOG entry

[MatMoore]

If we keep both regexes then we have 3 levels of support for this feature, which seems like it would complicate testing and responding to issues, even if 2 of them have the same behaviour

1. Doesn't support Intl.Segementer or the regex classes -> basic regex used by maxwords
2. Doesn't support Intl.Segmenter but does support the regex classes -> good regex
3. Supports Intl.Segmenter -> Intl.Segmenter

I think it would be fine to fall back to the basic regex whenever Intl.Segmenter is not available, rather than including a 3rd implementation. I.e. combine 1 & 2 into one implementation.

I might be wrong, but to me countType: 'words' feels like a soft limit that you might use if there is no hard constraint on the server side. So treating hyphenated words as one word probably won't break anything?

Another alternative might be to provide a polyfill that uses unicode ranges [\u0009-\u000D...] instead of the character classes, and use that as the fallback for Intl.Segmenter.

If we decide to have behaviour that differs across browsers we should probably document that wherever we introduce the feature.

[colinrotherham]

Thanks for looking at this @MatMoore

I've added a reply about the Babel regex transform over in #1899 (comment)

These are the browsers that shipped type="module" without Unicode character class escape support:

• Chrome 61–63
• Firefox 60–77
• Safari 11

We could simply upgrade the basic regex when opting in via countType: "words"?

- this.separator = /\s+/g;
+ this.separator = /[\s\-‑–—.,;:!\\/]+/g;

Whilst it was nice to use \p escapes (via the u flag) since so few browsers lacked support, this "good regex" is probably good enough for all other browsers lacking Intl.Segmenter

[MatMoore]

yeah that seems reasonable to me, it avoids the question of whether \p escapes are supported or not, and then when we drop maxwords we can get rid of the old regex.

[colinrotherham]

I've removed the various levels of regex (by browsers support) from this PR

This PR has been updated to apply the suggestion in alphagov/govuk-frontend#6995 (review) so that browsers without Intl.Segmenter fall back to the no-JS version

• Current options maxlength and maxwords work as usual for backwards compatibility
• New options countType: "characters" or countType: "words" use Intl.Segmenter

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
94.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud