Skip to content

CLDR_UPDATE_INTEGRATION.md

Latest commit

 

History

History
195 lines (138 loc) · 17.8 KB

CLDR_UPDATE_INTEGRATION.md

File metadata and controls

195 lines (138 loc) · 17.8 KB
Raw

CLDR Update Guide

This is the single authoritative guide for updating Localize to a new CLDR release. It covers the whole cycle: refreshing the vendored sources, regenerating locale and supplemental data, updating conformance code against the TR35 changes, verifying, and shipping. A new maintainer should be able to run an update end-to-end from this document alone.

Release-specific work items live in a per-release plan (plans/cldr-49.md for the October 2026 cycle); this guide is the repeatable process those plans hang off.

The model in one paragraph

At runtime Localize reads only compiled ETF files under priv/localize/ (plus Unicode Character Database tables under priv/unicode/). Those ETFs are generated by the build-time pipeline in data/ from raw CLDR sources vendored under priv/cldr/ (gitignored, reproducible from upstream). Only en.etf and und.etf locale files ship in the hex package; every other locale is downloaded at runtime from the CDN (Cloudflare R2, https://elixir-localize.com/locales/v<version>/<locale>.etf) and verified against a SHA-256 hash manifest bundled in the package (priv/localize/locale_hashes.etf). Updating CLDR means: refresh sources → regenerate ETFs → adapt code to spec changes → verify → upload to CDN → regenerate the hash manifest from the CDN bytes → release.

Directory layout

priv/
├── cldr/                          # Build-time source data (gitignored except essentials)
│   ├── bcp47/                     # BCP 47 extension key definitions (XML)
│   ├── collation/                 # Collation tailoring (XML) + FractionalUCA.txt
│   ├── external_sources/          # Script_Metadata.csv, iso_currencies.xml
│   ├── locales/<locale>/          # Per-locale JSON from CLDR production data
│   ├── supplemental_data/         # Supplemental JSON and XML
│   ├── validity/                  # Validity XML for subtag validation
│   └── FractionalUCA.txt          # Fractional UCA table (committed — collation + reorder data)
│
├── localize/                      # Runtime data (ETF) — committed except locales/
│   ├── locales/<locale>.etf       # Only en.etf and und.etf committed / packaged
│   ├── supplemental_data/*.etf
│   ├── validity/*.etf
│   ├── locale_hashes.etf          # SHA-256 manifest of CDN locale bytes (packaged)
│   ├── collation_table.etf, all_locale_names.etf, known_territories.etf, …
│   ├── version                    # CLDR version (e.g. "48.2")
│   └── localize_patch_version     # "{cldr_version}:{patch}" (e.g. "48.2:1")
│
├── unicode/                       # UCD tables (combining_class.txt, general_category.txt)

data/                              # Pipeline (compiles in :dev/:test only, not in the package)
├── data.ex                        # Entry points (Localize.Data.*)
├── locale.ex                      # Per-locale generation (incl. loader transforms)
├── locale_transformer.ex          # Struct transforms applied during generation
├── supplemental.ex, validity.ex, collation.ex, plural_rules.ex, …
├── normalize/                     # Per-domain JSON normalizers
└── mix/tasks/                     # Mix tasks (see reference table below)

scripts/
├── ldml2json                      # Legacy LDML→JSON wrapper (deprecated; misses annotations)
└── ldml2json_v2                   # Wraps unicode-org/cldr-json's cldr-generate-json.sh (use this)

Prerequisites

  • Toolchain. Elixir/OTP pinned by .mise.toml. Non-interactive shells do not pick up mise activation — prepend the pinned bin dirs to PATH before any mix command (see the toolchain section of ~/.claude/CLAUDE.md or mise which elixir).

  • CLDR repository — a checkout of github.com/unicode-org/cldr at the release tag. Location: $CLDR_REPO, default ../cldr_repo (on the primary workstation: ~/Development/cldr/cldr_repo).

  • CLDR production data — the pre-built JSON locale files, produced by scripts/ldml2json_v2 (requires Java/Maven; wraps cldr-generate-json.sh and emits all packages including annotations). Location: $CLDR_PRODUCTION_DATA, default ../cldr_production_data.

  • R2 credentials (upload/release phase only): R2_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, R2_BUCKET. In CI these are repository secrets used by .github/workflows/upload-locales.yml.

Mix task and script reference

Task / script Purpose
mix localize.update_cldr Orchestrates phases 1–3 (copy sources → generate supplemental → compile gate → generate locales → test gate), each step in a fresh VM. --check runs the preflight and prints the plan without changing anything; --locales en,fr trials a subset.
scripts/ldml2json_v2 Converts the CLDR XML repo to production JSON (all packages). Run once per CLDR drop.
mix localize.copy_sources Copies raw CLDR sources into priv/cldr/ and writes the CLDR version to priv/localize/version. --supplemental / --locales scope the copy.
mix localize.generate_supplemental Regenerates supplemental, validity, collation and top-level ETFs under priv/localize/.
mix localize.generate_locales Regenerates per-locale ETFs (priv/localize/locales/). Positional args scope to specific locales.
mix localize.generate_locale_hashes Regenerates the download-integrity manifest. Use --from-cdn after uploading so the manifest hashes the exact bytes consumers download (see "Hash manifest and the OTP encoding trap").
mix localize.update_mf2_conformance Re-vendors the 16 MessageFormat 2 working-group conformance files into test/support/data/mf2_conformance/. Run when the WG suite moves.
mix localize.bump_patch_version Bumps the Localize-side patch counter ({cldr_version}:{patch}); auto-resets to 0 when the CLDR version changes. Never bumped automatically.
mix localize.download_unicode_data Refreshes UCD tables in priv/unicode/ (own cadence — on Unicode releases, not every CLDR drop).
mix localize.download_iso_currencies Refreshes ISO 4217 data from SIX Group (own cadence).
mix localize.upload_locale Generates and uploads specific locales to R2 (--version required). For ad-hoc fixes; full releases go through the tag-triggered workflow.
.github/workflows/upload-locales.yml On version-tag push: regenerates all locales in CI and uploads to R2. Runs on a pinned CI OTP — its ETF bytes are the canonical CDN bytes.

The release runbook

Run the phases in order. Every phase ends with a verification gate; do not proceed past a red gate.

Phase 0 — Pre-release tracking (starts at CLDR alpha)

  1. Fork the CLDR release page (e.g. https://cldr.unicode.org/downloads/cldr-49) into plans/cldr-<N>-changes.md, one row per change, scored cosmetic / output-changing / API-affecting / data-only.
  2. Walk the translators' guide (https://cldr.unicode.org/translation) section by section into plans/cldr-<N>-translator-guide-checklist.md, marking each section correct / partial / missing / N/A against our implementation.
  3. Diff the conformance fixtures: diff -r $CLDR_REPO_old/common/testData $CLDR_REPO_new/common/testData --brief. For each delta: re-import updated fixtures we already ingest, add loaders for new ones in directories we cover, and log new directories as work items.
  4. Diff the TR35 spec chapters relevant to shipped features (dates, numbers, collation, messageFormat) and file work items for behavioural changes. After release, walk the TR35 §Modifications log into plans/cldr-<N>-retrospective.md (template: plans/cldr-48-retrospective.md) classifying every entry as data-only / code-applied / code-pending / skipped-with-reason.

Phases 1–3 in one command

Once the upstream checkouts are refreshed (the first two commands of Phase 1), the copy → generate → verify sequence can run as a single orchestrated task:

mix localize.update_cldr --check    # preflight: verify sources, report versions, print the plan
mix localize.update_cldr            # copy sources, generate supplemental (compile gate), generate locales (test gate)

The task stops after Phase 3 and prints the remaining manual phases. The sections below describe what it runs, for when a phase needs to be run or debugged individually.

Phase 1 — Refresh vendored sources

# 1. Update the upstream checkouts
(cd $CLDR_REPO && git fetch && git checkout release-<N>)
scripts/ldml2json_v2                     # regenerates $CLDR_PRODUCTION_DATA (Java/Maven)

# 2. Copy into the project (writes priv/localize/version)
mix localize.copy_sources

# 3. Auxiliary sources — only when their upstreams moved
mix localize.download_unicode_data      # UCD (Unicode release cadence)
mix localize.download_iso_currencies    # ISO 4217 (SIX Group cadence)
mix localize.update_mf2_conformance     # MF2 WG test suite (tracks the WG repo)

Also refresh priv/cldr/FractionalUCA.txt from the CLDR repo when the UCA version bumps — it feeds the collation table, the reorder-group data, and the variable-weight boundaries.

Gate: git status shows the expected source deltas and priv/localize/version carries the new CLDR version. The patch counter auto-resets to 0 at the next generation.

Phase 2 — Regenerate supplemental data

mix localize.generate_supplemental

Reads priv/cldr/{supplemental_data,validity,bcp47,collation}/ and writes the ETFs under priv/localize/. Every module that embeds this data at compile time declares it with @external_resource, so the regeneration automatically recompiles the timezone, time-preference, validity, script-mapping and collation modules — a stale-module bug class that existed before 0.46.0. Do not hand-edit or hand-generate any ETF; everything comes from the pipeline.

Gate: mix compile --warnings-as-errors then mix test. Failures here usually mean a CLDR data-shape change that a normalizer in data/normalize/ must accommodate — fix the normalizer, never the generated file.

Phase 3 — Regenerate locale data

mix localize.generate_locales            # all locales

Writes priv/localize/locales/<locale>.etf (only en.etf/und.etf are committed). The pipeline change-review rule: when any normalizer, @required_modules, or Localize.Data.Locale transform changes, regenerate all locales so the committed pair and the CDN set stay in step.

Attributing churn. ETF files are binary; a byte diff tells you nothing. To distinguish real data changes from encoding noise, compare decoded terms: load old and new ETFs with :erlang.binary_to_term/1 and diff the terms (a ~10-line script; the technique is called a semantic diff in past changelogs). From the October 2026 cycle onwards ETFs are encoded deterministically, so byte-identical means term-identical and this distinction matters less — but the semantic diff remains the tool for reviewing what the new CLDR version actually changed.

Gate: full mix test (the test helper downloads its 39 working locales from the freshly generated set in dev/test — they are generated on the fly, not downloaded, when the pipeline is newer).

Phase 4 — Conformance code updates

Data regeneration alone is not an update. Work through the Phase 0 artifacts:

  1. Spec-driven fixes. For each code-pending item from the changes log and translators' checklist, implement against the TR35 text (the authoritative checkout lives beside the repo: $CLDR_REPO/docs/ldml/tr35-*.md — always the file, never a possibly-stale web copy).
  2. Fixture-driven fixes. Wire the new/updated common/testData fixtures into test/ conformance suites and fix what they expose. Record the CLDR version + git SHA in each conformance file header.
  3. Parser/lexer changes. If the RBNF (or any) source text format changed, adapt src/*.xrl / *.yrl and the normalizers. Verify lexer changes with a token-stream diff: dump inspect(tokens) for every rule string in every locale with the old lexer, rebuild, dump again, diff — the streams must be identical unless the format change is intentional (harness precedent: the 0.46.0 rbnf_lexer fix validated 6,584 rules across 48 locales this way).
  4. MF2. If the WG suite was re-vendored, run the formatter conformance tests and triage new cases: implement, or add a documented per-case exclusion.

Gate: the full six-gate stack — mix format --check-formatted, mix compile --warnings-as-errors, mix test, mix credo --strict (0), mix dialyzer (0), MIX_ENV=release mix docs (0 warnings) — plus all conformance suites green.

Phase 5 — Ship: versioning, CDN, manifest, release (ORDER MATTERS)

  1. Update mix.exs @version, the CHANGELOG (a "CLDR base data" entry linking to plans/cldr-<N>-changes.md, plus user-visible output changes), and the README if the install snippet's ~> floor needs moving.

  2. Commit and push; tag the release (vX.Y.Z). The tag push triggers .github/workflows/upload-locales.yml, which regenerates all locales in CI and uploads them to R2 under the version prefix.

  3. After the upload completes, regenerate the hash manifest from the CDN:

    mix localize.generate_locale_hashes --from-cdn

    Commit the updated priv/localize/locale_hashes.etf. See the trap below for why this must hash CDN bytes and must happen after the upload.

  4. Run the release-review checklist (README install snippet, hexdocs links, executed doc examples, changelog claims) and mix hex.publish.

  5. Tag hygiene: the repo convention is a lightweight vX.Y.Z tag on the commit that was published.

Phase 6 — Post-release

  • Complete the modifications retrospective (plans/cldr-<N>-retrospective.md).
  • Roll unfinished items forward into the next release plan; date-stamp a closing entry in the current plan's change log.
  • Sweep plans/ quarterly for stale items.

Hash manifest and the OTP encoding trap

The download-integrity manifest (priv/localize/locale_hashes.etf, packaged in the hex release) maps locale → SHA-256 of the CDN object bytes. Localize.Locale.Provider verifies every download against it before decode; a mismatch fails closed with LocaleIntegrityError.

The trap: term_to_binary/1 output historically differed across OTP releases (map key ordering), and the CDN bytes are produced by CI's OTP (see the pinned version in upload-locales.yml), not your workstation's. Hashing locally generated files therefore produced a manifest that failed against the CDN — this broke CI in the 0.44.0 cycle and is why --from-cdn exists and why the manifest step comes after the upload. From October 2026 the pipeline encodes ETFs deterministically, making local and CI bytes identical — but keep using --from-cdn: it verifies what consumers actually download, which is the property the manifest exists to guarantee.

Hazards a new maintainer must know

  • Never write or delete through _build/*/lib/localize/priv — it is a symlink to the real priv/. An rm -rf "in the build directory" destroys source data (this has happened).
  • Never hand-generate or hand-edit an ETF. Data flows only through the pipeline tasks; a hand-built file will differ from regenerated output and poison the manifest or the semantic diff.
  • Do not run git state-changing commands as part of the update — the maintainer commits, tags, and pushes explicitly at the points the runbook calls out.
  • The dev/test locale set is generated on the fly; only en.etf/und.etf are committed. If local tests suddenly download from the CDN, your generated set is stale relative to priv/localize/version.
  • Patch releases within a CLDR version: pipeline changes without a CLDR bump use mix localize.bump_patch_version ({cldr_version}:{patch}), then regenerate + re-upload + re-manifest exactly as in Phase 5.

CLDR 49 (October 2026) one-time items

These land with the CLDR 49 cycle and then disappear from this guide. Detail and status live in plans/cldr-49.md and the task list.

  1. Deterministic ETF encoding. Encode all generated ETFs with deterministic map ordering (e.g. term_to_binary(term, deterministic: true)) so regeneration is byte-stable across OTP releases. Accept the one-time full-file churn in the same commit as the CLDR 49 data churn.
  2. Collation ETF: reorder-group starts. Emit the FDD1 group-marker fractions (SPACE/PUNCTUATION/SYMBOL/CURRENCY/DIGIT first-primary {lead, sub} bytes) from FractionalUCA.txt into the collation ETF; read them in Localize.Collation.Variable.primary_range/1 and delete the pinned @group_starts constants there.
  3. Remove the :alt shim. The pipeline fix for the "alt" (Southern Altai) language-code atomization is already in data/locale.ex (restore_alt_language_code/1); once CLDR 49 data is regenerated, delete fix_alt_language_key/1 from Localize.Locale.Provider.PersistentTerm.
  4. Adopt scripts/ldml2json_v2 as the only documented conversion path; deprecate and later remove scripts/ldml2json (v2 emits the annotations packages that the emoji work needs).
  5. RBNF source-format change. CLDR 49 reshapes RBNF rule text (including the remove rule from ruleset mechanic). Adapt data/normalize/rbnf.ex (and the lexer/parser if the rule text itself changed — use the token-diff harness), keeping the runtime ETF shape stable where possible.
  6. Remaining feature items from plans/cldr-49.md: semantic skeletons (child plan plans/semantic-skeletons.md), date-time append items (needs the new dateFields ingestion), minimal pairs, min/max significant digits (gated on the CLDR 49 decimal fixtures), the common/testData audit, and the two 48-retrospective follow-ups (gmtUnknownFormat consumer, FractionalUCA_blanked.txt ingestion).

Item 13 of the original CLDR 49 plan (CDN checksum manifests) shipped early, in Localize 0.44.0, as the hash-manifest system described above.