Hugo cannot identity correct translation string when multiple variants of a language code is used

[shaform]

What version of Hugo are you using (hugo version)?

$ hugo version
Hugo Static Site Generator v0.78.2 linux/amd64 BuildDate: 2020-11-13T14:14:29Z

Does this issue reproduce with the latest release?

Yes.

Steps to Reproduce

The problem can be reproduced by following the steps below:

1. Copy the current multilingual example to local directory: https://github.com/gohugoio/hugo/tree/b8758de19ec75b4565075314f9578270a092bc6f/examples/multilingual

2. cd into the local directory and create zh files by executing the commands:

cp content/home.en.md content/home.zh.md
cp i18n/en.toml i18n/zh.toml
cp i18n/en.toml i18n/zh-cn.toml

3. Modify i18n/zh.toml, set other= Correct title

4. Modify i18n/zh-cn.toml, set other= Wrong title

5. Add the following content into config.toml

[languages.zh]
weight = 1
title = "Test ZH language"
[[languages.zh.menu.main]]
url = "/zh/home"
name = "ZH Home"
weight = 0

6. Execute hugo server.
   When hugo server is executed, a warning message is displayed, indicating something is wrong:

WARN 2020/11/24 10:10:59 Failed to get translated string for language "zh" and ID "head_title": %!s(<nil>)

7. Open browser and go to url HUGO_URL/zh/home. We should expect that the page title is Correct title. However, Wrong title is displayed as the page title.

Comments

This is probably related to #7838.