Skip to content

PRO-824: Use Unicode-aware regex in ambiguous text check #1730

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SteveJonesDev wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

PRO-824: Use Unicode-aware regex in ambiguous text check #1730

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions src/pageScanner/checks/has-ambiguous-text.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,16 @@ const ambiguousPhrases = [
__( 'opens a new window', 'accessibility-checker' ),
];

const normalizedPhrases = ambiguousPhrases.map(
( p ) => p.toLowerCase().replace( /[^\p{L}]+/gu, ' ' ).trim()
);
Comment on lines +26 to +28

@gemini-code-assist gemini-code-assist Bot May 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using only \p{L} (Letters) in the Unicode regex will strip combining diacritical marks (\p{M}), which are used in many languages (e.g., Hindi, Arabic, Thai, or even Latin languages when strings are in NFD/decomposed form, such as text copied from macOS). Stripping these marks will break words into fragments and cause matching to fail.

To prevent this, we should:

  1. Include \p{M} in the character class: /[^\p{L}\p{M}]+/gu.
  2. Use .normalize() to ensure consistent Unicode normalization (NFC) before processing.
Suggested change
const normalizedPhrases = ambiguousPhrases.map(
( p ) => p.toLowerCase().replace( /[^\p{L}]+/gu, ' ' ).trim()
);
const normalizedPhrases = ambiguousPhrases.map(
( p ) => p.normalize().toLowerCase().replace( /[^\p{L}\p{M}]+/gu, ' ' ).trim()
);


const checkAmbiguousPhrase = ( text ) => {
if ( ! text ) {
return false;
}
text = text.toLowerCase().replace( /[^a-z]+/g, ' ' ).trim();
return ambiguousPhrases.includes( text );
text = text.toLowerCase().replace( /[^\p{L}]+/gu, ' ' ).trim();

@chatgpt-codex-connector chatgpt-codex-connector Bot May 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve combining marks when normalizing Unicode text

For locales with accented translations, this drops combining marks instead of treating canonically equivalent text the same. For example, the Vietnamese translation for “continue” is tiếp tục; if the DOM contains the same visible text in decomposed form from copy/paste, textContent normalizes to something like tie p tu c while the precomposed translated phrase normalizes to tiếp tục, so the ambiguous link is missed. Include marks in the character class or normalize both strings to the same Unicode form before stripping punctuation.

Useful? React with 👍 / 👎.

return normalizedPhrases.includes( text );
Comment on lines +34 to +35

@gemini-code-assist gemini-code-assist Bot May 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Apply the same Unicode-aware normalization (including \p{M} and .normalize()) to the input text to ensure consistent matching with the normalized phrases.

Suggested change
text = text.toLowerCase().replace( /[^\p{L}]+/gu, ' ' ).trim();
return normalizedPhrases.includes( text );
text = text.normalize().toLowerCase().replace( /[^\p{L}\p{M}]+/gu, ' ' ).trim();
return normalizedPhrases.includes( text );

};

export default {
Expand Down
Loading