Create "Compare" And "More" Pages In Wagtail (#1241)

* update get_locales_for_cms_page() to also return fallback locale codes

* add a utility function to find the fallback page for a locale

* add a migration to ensure expected Locales exist in database

* update FALLBACK_LOCALES mapping in settings to match expected locale fallback pattern

* update CMSLocaleFallbackMiddleware to serve content from a fallback locale when the requested locale page is not found

* update SpringfieldLocale.get_active to find fallback locale if needed

Note: this logic shouldn't be needed, since all of
the locales should be present in the database, but
in case one of the locale records is removed from
the database, but the locale is still referenced
in the code (for example, in the settings), this
logic will handle it.

* update i18n context processor to return a CANONICAL_LANG

CANONICAL_LANG matches LANG, unless a page is served
for a fallback locale. In this case, the template
can determine which context variable (LANG or
CANONICAL_LANG) to use in each relevant place.

* update SpringfieldLinkBlock to return expected page URLs

when a user requests a page and is given content
for a fallback locale, the page links in the
content should match the requested URL's locale.

* avoid bug of always returning a 200 response by reverting commit 356d331

* add tests for changes to get_locale() and render()

* make sure that locales created in migration are translations of root page

* update get_locale_options to return alias locales that have fallback locales

* make sure canonical href points to appropriate URL for content

* add a test that alias locale pages do not appear in the sitemap

* for consistency, make sure non-Wagtail pages also serve fallback content from alias locale URL when alias locale has no content

* serve fallback locale content even when alias Locale does not exist in the database

* make sure page links in link block return alias locale URLs even when alias Locale does not exist in the database

* update tests that rely on FALLBACK_LOCALES to explicitly set it

* correct an incorrectly defined variable in a test

* make sure relative links in link block return alias locale URLs even when alias Locale does not exist in the database

* add an indicator for alias locales on the locale list page in Wagtail

* make sure root pages in new Locales are properly created

this fixes an error where translating a page into
the new Locales was causing a server error,
because wagtail-localize was not able to correctly
find the root page in the new Locale.

* make sure page links correctly point to page URLs in other locales

* add column filters in the wagtail localize dashboard

* use latest version of wagtail-localize-dashboard

* make sure canonical and alternate hreflangs are valid

do not show hreflang alternates for alias locales
that don't have their own content

* add template file used in test

* avoid leaking test state by modifying jinja2_loader.searchpath, rather than setting template engines

* make sure new root pages are created at the correct depth

* make sure we correctly render homepage content for alias locales

* make logic more readable by separating middleware and URL-routing logic

The separation:
 - makes the logic more readable
 - avoids a scenario where the middleware intercepts
   the request and returns a Wagtail page when it
   should not

* reorder migrations after merge

* make sure redirects based on Accept-Language header work for non-depth-2 home page sites

Previously, the middleware's Accept-Language
redirect was hardcoding Wagtail url_path prefixes
as /home and /home-{locale}, which only works when
the site root page is at depth 2. In environments
where Site.root_page is at a different depth (for
example at depth 3 with url_path=/home/home/), the
hardcoded prefixes were not matching pages,
causing 404s instead of redirects for Wagtail
Locales that did not have a specific page.

* reorder migrations after merge

* fix minor typos in tests

* add unit tests for different root page scenarios in fallback locale

* disable creating alias locale records in db export script

* update comment in migration to be accurate based on middleware refactoring

* make code more readable by referring to request interception in a consistent way

* make code more readable by moving fallback locale check into helper function

* add explicit test that a live Page in an alias locale is served as expected

* be more specific about the exception when the site root does not exist in a fallback locale

* be more efficient when calculating existing locales

* be more specific about the exceptions raised in SpringfieldLinkBlockURLValue.get_url()

* Remove extra whitespace in comment

Co-authored-by: Steve Jalim <stevejalim@mozilla.com>

* get locale directly from the URL, rather than an attribute on the request

This change makes the code less likely to break
in case any changes are made to our code that
sets the 'locale' attribute on the request

* update 'compare' and 'more' URLs to use prefer_cms()

* add helper functions for FTL parsing

* add a field to determine if article detail siblings should be visible on article index page

* add a management command to create 'compare' and 'more' CMS pages

* add migrations to create 'compare' and 'more' pages

* reorder migrations after merge

* reorder migrations after merge

* explicitly set show_sibling_detail_pages in test object

* update 'compare' and 'more' URLs to use prefer_cms()

* add helper functions for FTL parsing

* add a field to determine if article detail siblings should be visible on article index page

* add a management command to create 'compare' and 'more' CMS pages

* add migrations to create 'compare' and 'more' pages

* explicitly set show_sibling_detail_pages in test object

* reorder migrations to account for changes in main branch

* reorder migrations after merge

* reorder migrations after merge

* reorder migrations after merge

* [reset-db]

* [reset-db]

* add wagtailsearch migration dependency, to allow migration to run on fresh databases

* Organize imports on create compare pages command

* ArticleIndexPage supports 3 different card types

* update logic to create 'compare' and 'more' pages to set pages' index_card_type

* merge migration

* updated migration list

---------

Co-authored-by: Steve Jalim <stevejalim@mozilla.com>
Co-authored-by: Kasey Kelly <kasey@servee.com>
Co-authored-by: Mariana Bedran Lesche <maribedran@gmail.com>

Author: 4 people

springfield/cms/fixtures/compare_more_page_fixtures.py

@@ -17,29 +17,105 @@
 
 from django.conf import settings
 
+import polib
 from fluent.syntax import parse
 from fluent.syntax.ast import Message, Placeable, TextElement
+from wagtail_localize.models import StringSegment
 
 # Brand terms to expand in FTL strings.
 # These match the terms defined in l10n/en/brands.ftl
 BRAND_TERMS = {
+    # Company names
+    "-brand-name-apple": "Apple",
+    "-brand-name-creative-commons": "Creative Commons",
+    "-brand-name-facebook": "Facebook",
+    "-brand-name-google": "Google",
+    "-brand-name-microsoft": "Microsoft",
+    "-brand-name-mozilla": "Mozilla",
+    "-brand-name-mozilla-corporation": "Mozilla Corporation",
+    "-brand-name-mozilla-foundation": "Mozilla Foundation",
+    "-brand-name-netscape": "Netscape",
+    "-brand-name-linkedin": "LinkedIn",
+    "-brand-name-spotify": "Spotify",
+    # Firefox browsers
     "-brand-name-firefox": "Firefox",
+    "-brand-name-firefox-beta": "Firefox Beta",
     "-brand-name-firefox-browser": "Firefox Browser",
-    "-brand-name-mozilla": "Mozilla",
+    "-brand-name-firefox-browsers": "Firefox Browsers",
+    "-brand-name-firefox-developer-edition": "Firefox Developer Edition",
+    "-brand-name-firefox-enterprise": "Firefox Enterprise",
+    "-brand-name-firefox-esr": "Firefox ESR",
+    "-brand-name-firefox-extended-support-release": "Firefox Extended Support Release",
+    "-brand-name-firefox-focus": "Firefox Focus",
+    "-brand-name-firefox-nightly": "Firefox Nightly",
+    # Firefox browsers (short names)
+    "-brand-name-beta": "Beta",
+    "-brand-name-developer-edition": "Developer Edition",
+    "-brand-name-enterprise": "Enterprise",
+    "-brand-name-esr": "ESR",
+    "-brand-name-focus": "Focus",
+    "-brand-name-nightly": "Nightly",
+    # Firefox browsers (legacy)
+    "-brand-name-firefox-aurora": "Firefox Aurora",
+    # Firefox products
+    "-brand-name-facebook-container": "Facebook Container",
+    "-brand-name-firefox-devtools": "Firefox DevTools",
+    "-brand-name-firefox-relay": "Firefox Relay",
+    "-brand-name-firefox-translations": "Firefox Translations",
+    # Firefox products (short names)
+    "-brand-name-relay": "Relay",
+    # Firefox projects
+    "-brand-name-firefox-labs": "Firefox Labs",
+    # Pocket
+    "-brand-name-pocket": "Pocket",
+    # Fakespot
+    "-brand-name-fakespot": "Fakespot",
+    # Mozilla projects
+    "-brand-name-bugzilla": "Bugzilla",
+    "-brand-name-gecko": "Gecko",
+    "-brand-name-mdn-plus": "MDN Plus",
+    "-brand-name-mdn-web-docs": "MDN Web Docs",
+    "-brand-name-mozilla-monitor": "Mozilla Monitor",
+    "-brand-name-mozilla-vpn": "Mozilla VPN",
     "-brand-name-mozilla-account": "Mozilla account",
     "-brand-name-mozilla-accounts": "Mozilla accounts",
-    "-brand-name-gecko": "Gecko",
-    "-brand-name-ipad": "iPad",
-    "-brand-name-iphone": "iPhone",
-    "-brand-name-firefox-translations": "Firefox Translations",
+    "-brand-name-mozilla-ai-v2": "Mozilla.ai",
+    "-brand-name-mozilla-ventures": "Mozilla Ventures",
+    "-brand-name-thunderbird": "Thunderbird",
+    # Mozilla projects (short names)
+    "-brand-name-common-voice": "Common Voice",
+    "-brand-name-mdn": "MDN",
+    "-brand-name-monitor": "Monitor",
+    # Other browsers
+    "-brand-name-brave": "Brave",
     "-brand-name-chrome": "Chrome",
     "-brand-name-edge": "Edge",
+    "-brand-name-ie": "Internet Explorer",
+    "-brand-name-opera": "Opera",
     "-brand-name-safari": "Safari",
+    # Platforms
     "-brand-name-android": "Android",
+    "-brand-name-chromeos": "Chrome OS",
     "-brand-name-ios": "iOS",
-    "-brand-name-windows": "Windows",
-    "-brand-name-macos": "macOS",
     "-brand-name-linux": "Linux",
+    "-brand-name-mac": "macOS",
+    "-brand-name-mac-short": "Mac",
+    "-brand-name-macos": "macOS",
+    "-brand-name-windows": "Windows",
+    # Apple products
+    "-brand-name-app-store": "App Store",
+    "-brand-name-ipad": "iPad",
+    "-brand-name-iphone": "iPhone",
+    "-brand-name-test-flight": "TestFlight",
+    # Facebook products
+    "-brand-name-instagram": "Instagram",
+    # Google products
+    "-brand-name-chromebook": "Chromebook",
+    "-brand-name-chromium": "Chromium",
+    "-brand-name-google-play": "Google Play",
+    "-brand-name-youtube": "YouTube",
+    # Enterprise
+    "-brand-name-support-for-organizations": "Support for Organizations",
 }
 
 
@@ -195,6 +271,36 @@ def get_english_ftl_strings(ftl_filename: str) -> dict[str, str]:
     return parse_ftl_file(en_path)
 
 
+def get_ftl_path_for_locale_at_subpath(locale: str, ftl_relative_path: str) -> Path | None:
+    """Like get_ftl_path_for_locale but accepts a full relative path.
+
+    Args:
+        locale: e.g. "de"
+        ftl_relative_path: e.g. "firefox/browsers/compare/brave.ftl"
+    """
+    external_path = settings.FLUENT_REPO_PATH / locale / ftl_relative_path
+    if external_path.exists():
+        return external_path
+    local_path = settings.FLUENT_LOCAL_PATH / locale / ftl_relative_path
+    if local_path.exists():
+        return local_path
+    return None
+
+
+def get_ftl_translations_at_subpath(locale: str, ftl_relative_path: str) -> dict[str, str]:
+    """Get translations for a specific locale using a full relative FTL path."""
+    ftl_path = get_ftl_path_for_locale_at_subpath(locale, ftl_relative_path)
+    if ftl_path:
+        return parse_ftl_file(ftl_path)
+    return {}
+
+
+def get_english_ftl_strings_at_subpath(ftl_relative_path: str) -> dict[str, str]:
+    """Get English source strings using a full relative FTL path."""
+    en_path = settings.FLUENT_LOCAL_PATH / "en" / ftl_relative_path
+    return parse_ftl_file(en_path)
+
+
 def get_ui_ftl_path_for_locale(locale: str) -> Path | None:
     """
     Get the path to ui.ftl for a specific locale.
@@ -362,6 +468,48 @@ def convert_ftl_links_to_wagtail(translated: str, source: str) -> str:
     return result
 
 
+def build_po_from_ftl(translation, en_text_to_msgid: dict[str, str], translations: dict[str, str]):
+    """Build a PO file by matching Wagtail segments to FTL translations.
+
+    Args:
+        translation: A wagtail_localize Translation instance.
+        en_text_to_msgid: Mapping of normalized English text to FTL message IDs
+            (built with build_text_to_msgid_mapping).
+        translations: Mapping of FTL message IDs to translated text for the target locale.
+
+    Returns:
+        A polib.POFile populated with matched translation entries.
+    """
+    po = polib.POFile(wrapwidth=200)
+    po.metadata = {
+        "MIME-Version": "1.0",
+        "Content-Type": "text/plain; charset=utf-8",
+        "X-WagtailLocalize-TranslationID": str(translation.uuid),
+    }
+
+    string_segments = StringSegment.objects.filter(source=translation.source).select_related("string", "context")
+
+    for segment in string_segments:
+        source_text = segment.string.data
+        normalized_source = normalize_text_for_matching(source_text)
+
+        msgid = en_text_to_msgid.get(normalized_source)
+
+        if msgid and msgid in translations:
+            translated_text = translations[msgid]
+            translated_text = convert_ftl_links_to_wagtail(translated_text, source_text)
+
+            po.append(
+                polib.POEntry(
+                    msgid=source_text,
+                    msgctxt=segment.context.path,
+                    msgstr=translated_text,
+                )
+            )
+
+    return po
+
+
 def build_text_to_msgid_mapping(ftl_strings: dict[str, str]) -> dict[str, str]:
     """
     Build a mapping from normalized text to FTL message ID.