useExtracted auto-generated message keys starting with `/` cause Google crawler 404 errors

[coobeet]

Description

When using useExtracted with auto-generated message keys, some keys begin with the / character. Google's crawl bot interprets these as URL paths and attempts to crawl them, resulting in Not Found (404) errors reported in Google Search Console.

This is because the auto-generated key character set includes /, which is a valid URL path character. When these keys appear in the rendered HTML or page source, Googlebot treats them as relative URLs.

Screenshot from Google Search Console:

Verifications

• I've read the docs. In particular, I've looked at the relevant usage guide.
• I've searched for existing issues and discussions on GitHub and Stack Overflow.
• I've compared my setup to a working example.

Mandatory reproduction URL

This issue is not reproducible in a code sandbox — it manifests in Google Search Console when Googlebot crawls a production site using useExtracted with auto-generated keys. The auto-generated keys (e.g. ones starting with /) appear in the page source and Googlebot interprets them as relative URL paths, then reports 404 errors.

Reproduction description

1. Use useExtracted with auto-generated message keys in a Next.js app
2. Deploy to production and let Google crawl the site
3. Open Google Search Console → Pages → Not Found (404)
4. Observe 404 entries corresponding to auto-generated message keys that start with /

Expected behaviour

Auto-generated message keys should not use characters that can be interpreted as URL paths by web crawlers. The character set should exclude URL-significant characters like /, ?, #, &.

Suggestion:

• Use a character set for auto-generated keys that excludes URL-like characters, OR
• Allow users to customize the character set / prefix used for auto-generated keys

[amannn]

Two questions here:

1. Do you render generated keys as-is to the page markup? They are intended to be replaced by actual text from a given locale.
2. Where do you render the keys? In a link? Even if a fallback is shown somewhere, what makes Googlebot believe this is a link?

On a separate note, I did consider not using the characters you mentioned above, but ultimately left them in for compatibility for react-intl users migrating to next-intl (they use the same key schema). My consideration was more around people who build translation management systems though and want to deeplink to a particular message—for these cases, you'd have to use encodeURIComponent on message keys when intending to put them into a URL.

Some example code or a repro would be helpful to better understand your use case.

[coobeet]

Thanks for the quick reply!

1. No, I don't render the generated keys as-is. t() always returns the resolved translation value. All my usage follows the standard pattern like {t("Some text")} in JSX.

2. The keys aren't rendered in any link element or visible text. However, the message keys are serialized into the page source as part of the data payload that NextIntlClientProvider passes from server to client (similar to __NEXT_DATA__). Googlebot is known to extract anything resembling a path (starting with /) from these serialized payloads and treat them as internal links to crawl — this is a documented behavior in vercel/next.js#40143.

For reference, in my project 56 out of 3,361 auto-generated keys start with / (e.g., /qP5M+, /4+wm2, /T1ODU), which matches the 404 entries I see in Google Search Console.

I understand the react-intl compatibility rationale for the character set. Perhaps the simplest fix would be to ensure the hash generation avoids / as a leading character (or URL-significant characters generally), while maintaining compatibility for the rest of the key?

[amannn]

this is a documented behavior in vercel/next.js#40143.

Interesting, I was really not aware of that!

I also found these related conversations:

• Generate message id should be url safe lingui/js-lingui#1789
• Generate Lingui ID with URL and filename safe alphabet lingui/js-lingui#2207
• feat!: use base64 urlsafe alphabet for id lingui/js-lingui#2438

On the react-intl repo there's somehow no mention of this issue.

Perhaps the simplest fix would be to ensure the hash generation avoids / as a leading character (or URL-significant characters generally), while maintaining compatibility for the rest of the key?

That would unfortunately still be breaking.

But, after reading vercel/next.js#40143 I understand that this is something that might be an issue for many users, with no really good workaround currently.

useExtracted is still experimental and I think there would be a possibility for changing the key hash in case this is has strong benefits.

I've linked this issue into #2087 to potentially change the behavior and will give it a bit more thought.

Thanks for raising this!