Skip to content

Commit ffd698f

Browse files
committed
Rename to localize_sql; add tagged-decimal aggregates and Ecto serialization for all Localize types
1 parent f5711f0 commit ffd698f

47 files changed

Lines changed: 5243 additions & 70 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/ci.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -149,7 +149,7 @@ jobs:
149149
--health-retries 5
150150
151151
env:
152-
LOCALIZE_ECTO_SQLITE_ICU: "true"
152+
LOCALIZE_SQL_SQLITE_ICU: "true"
153153

154154
steps:
155155
- name: Checkout
@@ -183,7 +183,7 @@ jobs:
183183
- name: Verify the extension was built
184184
run: test -f priv/localize_icu.so
185185

186-
# test_helper raises when LOCALIZE_ECTO_SQLITE_ICU is set but the
186+
# test_helper raises when LOCALIZE_SQL_SQLITE_ICU is set but the
187187
# extension cannot be found, so this cannot pass with the SQLite
188188
# suites silently excluded.
189189
- name: Run tests including the SQLite suites
@@ -199,7 +199,7 @@ jobs:
199199
runs-on: macos-latest
200200

201201
env:
202-
LOCALIZE_ECTO_SQLITE_ICU: "true"
202+
LOCALIZE_SQL_SQLITE_ICU: "true"
203203

204204
steps:
205205
- name: Checkout

.gitignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ erl_crash.dump
2020
*.ez
2121

2222
# Ignore package tarball (built via "mix hex.build").
23-
localize_ecto-*.tar
23+
localize_sql-*.tar
2424

2525
# Build artifacts of the localize_icu SQLite extension. The sources in
2626
# c_src/ are checked in and shipped; what the compiler produces is not.

CHANGELOG.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,22 @@ All notable changes to this project will be documented in this file.
44

55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
66

7+
## [Unreleased]
8+
9+
Renamed from `localize_ecto` to **`localize_sql`** — the package is about SQL databases (collations, serialization, migrations), not Ecto in general, which can be used without a database. The `Localize.Ecto.*` module namespace and the `mix localize.ecto.audit` task are unchanged; the app, config key (`:localize_sql`), env var (`LOCALIZE_SQL_SQLITE_ICU`) and `priv/localize_sql/` path are renamed.
10+
11+
`localize_sql` grows from locale-aware collation into the SQL home for every Localize type — serializing the full set of Localize and Elixir data types and generating the tagged-decimal aggregate machinery `ex_money_sql` builds on.
12+
13+
### Added
14+
15+
* `Localize.Ecto.TaggedDecimal` and its `DDL` generate a PostgreSQL composite type with tag-guarded `sum`/`avg`/`min`/`max` aggregates and `+`/`-` operators for any tag-plus-decimal value; a mismatched tag (euros plus yen) raises in the database. `ex_money_sql` now builds `money_with_currency` on this rather than its own SQL.
16+
17+
* Unit serialization, folded in from `localize_units_sql`: `Localize.Unit.Ecto.{Composite,Map}.Type` and the `Localize.UnitWithUsage` variants, `Localize.Unit.DDL` (now `sum`/`min`/`max`/`avg` for `cldr_unit` and `cldr_unit_with_usage`), and `mix localize.unit.gen.migration`.
18+
19+
* Ecto types for the remaining data types: `Localize.Currency.Ecto.Type`, `Localize.LanguageTag.Ecto.Type`, `Localize.Territory.Ecto.Type` and `Localize.Script.Ecto.Type` (text); `Localize.Ecto.Type.IntegerRange` and `.DateRange` (`int8range`/`daterange`); `Localize.Duration.Ecto.Type` and `Localize.Ecto.Type.Duration` (`interval`).
20+
21+
* `Localize.Ecto.Migration.Generator` renders and writes migrations for the SQL-generating mix tasks.
22+
723
## [1.0.0-rc.0] - 2026-07-28
824

925
The 1.0 release candidate. `localize_ecto` now supports SQLite as well as PostgreSQL, with the same locale resolving to the same collation on both.

README.md

Lines changed: 65 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
1-
# Localize Ecto
1+
# Localize SQL
22

3-
Locale-aware ICU collation for [Ecto](https://hexdocs.pm/ecto) queries on PostgreSQL and SQLite. `localize_ecto` resolves a [Localize](https://hexdocs.pm/localize) language tag to the best matching ICU collation and applies it with a `COLLATE` clause, so query results sort and compare according to the conventions of the user's locale.
3+
SQL database support for [Localize](https://hexdocs.pm/localize) and [Ecto](https://hexdocs.pm/ecto), in two parts: **locale-aware ICU collation** for query ordering, and **Ecto types** that store every Localize and relevant Elixir data type — money and units of measure with database-side aggregates, currencies, locales, territories, ranges, durations and more.
4+
5+
The first part resolves a Localize language tag to the best matching ICU collation and applies it with a `COLLATE` clause, so query results sort and compare according to the conventions of the user's locale:
46

57
```elixir
68
import Ecto.Query
@@ -19,9 +21,9 @@ from p in Product, where: collate(p.name < "münchen", "de")
1921
from p in Product, order_by: collate(p.name, collation: "german_phonebook")
2022
```
2123

22-
The examples above are PostgreSQL. `Localize.Ecto` exposes the PostgreSQL macros directly; for SQLite, import [Localize.Ecto.SQLite3](https://hexdocs.pm/localize_ecto/Localize.Ecto.SQLite3.html) instead — see [SQLite](#sqlite) below.
24+
The examples above are PostgreSQL. `Localize.Ecto` exposes the PostgreSQL macros directly; for SQLite, import [Localize.Ecto.SQLite3](https://hexdocs.pm/localize_sql/Localize.Ecto.SQLite3.html) instead — see [SQLite](#sqlite) below.
2325

24-
Locale resolution uses the [CLDR Language Matching](https://www.unicode.org/reports/tr35/tr35.html#LanguageMatching) algorithm, so any valid locale finds its best available collation — `"zh-TW"` resolves to `zh-Hant-x-icu`, `"de-DE"` to `de-x-icu`, and an unknown locale falls back gracefully. Locales that carry a BCP 47 collation type, such as `de-u-co-phonebk` (German phonebook order), resolve to collations you create once in a migration with [Localize.Ecto.Migration.create_collation/2](https://hexdocs.pm/localize_ecto/Localize.Ecto.Migration.html#create_collation/2).
26+
Locale resolution uses the [CLDR Language Matching](https://www.unicode.org/reports/tr35/tr35.html#LanguageMatching) algorithm, so any valid locale finds its best available collation — `"zh-TW"` resolves to `zh-Hant-x-icu`, `"de-DE"` to `de-x-icu`, and an unknown locale falls back gracefully. Locales that carry a BCP 47 collation type, such as `de-u-co-phonebk` (German phonebook order), resolve to collations you create once in a migration with [Localize.Ecto.Migration.create_collation/2](https://hexdocs.pm/localize_sql/Localize.Ecto.Migration.html#create_collation/2).
2527

2628
## Beyond ordering
2729

@@ -48,7 +50,7 @@ field :time_zone, Localize.Ecto.Type.TimeZone
4850
from e in Event, select: at_time_zone(e.starts_at, "Australia/Sydney")
4951
```
5052

51-
The [audit task](https://hexdocs.pm/localize_ecto/Mix.Tasks.Localize.Ecto.Audit.html) reports collation version drift after PostgreSQL/ICU upgrades — with the `REINDEX` and `ALTER COLLATION … REFRESH VERSION` remediation for every dependent index — and compares the server's Unicode, ICU and time zone inventories against the application's:
53+
The [audit task](https://hexdocs.pm/localize_sql/Mix.Tasks.Localize.Ecto.Audit.html) reports collation version drift after PostgreSQL/ICU upgrades — with the `REINDEX` and `ALTER COLLATION … REFRESH VERSION` remediation for every dependent index — and compares the server's Unicode, ICU and time zone inventories against the application's:
5254

5355
```console
5456
$ mix localize.ecto.audit
@@ -60,34 +62,78 @@ Audit for MyApp.Repo
6062
Time zones: all application zones known to the server (100 server-only zones)
6163
```
6264

65+
## Serializing Localize types
66+
67+
The second part is an `Ecto.Type` for every Localize and relevant Elixir data type, so a locale, a currency, a unit of measure or a range is a first-class value in a schema. Most store as ordinary text or `jsonb` and need no migration beyond the column:
68+
69+
```elixir
70+
schema "listings" do
71+
field :locale, Localize.LanguageTag.Ecto.Type # text, e.g. "en-GB"
72+
field :currency, Localize.Currency.Ecto.Type # text, e.g. "USD"
73+
field :country, Localize.Territory.Ecto.Type # text, e.g. "US"
74+
field :levels, Localize.Ecto.Type.IntegerRange # int8range, e.g. 1..10
75+
field :period, Localize.Ecto.Type.DateRange # daterange
76+
field :retention, Localize.Ecto.Type.Duration # interval
77+
end
78+
```
79+
80+
A value cast from user input is validated and canonicalized on the way in — an unknown currency or an invalid locale is rejected, and `"en_US"` becomes the resolved `"en-US"` language tag — and loaded back as the rich Localize value rather than the raw string.
81+
82+
### Money and units: database-side aggregates
83+
84+
A money amount or a unit of measure is a *tagged decimal* — a decimal paired with a tag (a currency code, a unit name) that is meaningless without it. `localize_sql` stores these as a PostgreSQL composite type and generates tag-guarded `sum`, `avg`, `min` and `max` aggregates and `+`/`-` operators, so the database itself refuses to add euros to yen or metres to feet:
85+
86+
```elixir
87+
# in a migration
88+
Localize.Unit.DDL.execute_each(Localize.Unit.DDL.create_cldr_unit())
89+
Localize.Unit.DDL.execute_each(Localize.Unit.DDL.define_aggregate_functions())
90+
91+
# summing a column of like units works; mixing units raises in the database
92+
Repo.aggregate(from(m in Measurement), :sum, :distance)
93+
```
94+
95+
The same machinery, [Localize.Ecto.TaggedDecimal](https://hexdocs.pm/localize_sql/Localize.Ecto.TaggedDecimal.html), backs [ex_money_sql](https://hexdocs.pm/ex_money_sql), which builds its `money_with_currency` type and aggregates on `localize_sql` rather than defining its own.
96+
97+
| Type | Ecto type | Column |
98+
|---|---|---|
99+
| Currency | `Localize.Currency.Ecto.Type` | text |
100+
| Language tag | `Localize.LanguageTag.Ecto.Type` | text |
101+
| Territory / Script | `Localize.Territory.Ecto.Type` / `Localize.Script.Ecto.Type` | text |
102+
| Time zone | `Localize.Ecto.Type.TimeZone` | text |
103+
| Unit of measure | `Localize.Unit.Ecto.Composite.Type` (also `Map`, and the `WithUsage` variants) | composite / jsonb |
104+
| Integer / date range | `Localize.Ecto.Type.IntegerRange` / `.DateRange` | int8range / daterange |
105+
| Duration | `Localize.Duration.Ecto.Type` and `Localize.Ecto.Type.Duration` | interval |
106+
107+
The composite, range and duration types map to PostgreSQL types (via `postgrex`) and light up only when it is present; the text and `jsonb` types work on any Ecto adapter.
108+
63109
## Installation
64110

65-
Add `localize_ecto` to your dependencies:
111+
Add `localize_sql` to your dependencies:
66112

67113
```elixir
68114
def deps do
69115
[
70-
{:localize_ecto, "~> 1.0-rc"}
116+
{:localize_sql, "~> 1.0-rc"}
71117
]
72118
end
73119
```
74120

75121
## SQLite
76122

77-
SQLite has no ICU collations of its own, so `localize_ecto` ships one: `localize_icu`, a SQLite loadable extension that registers ICU collations under the same names PostgreSQL uses. The same locale resolves to the same collation name and produces the same ordering on either database, so a query ported between them sorts identically — the test suite asserts this for a range of languages and tailorings.
123+
SQLite has no ICU collations of its own, so `localize_sql` ships one: `localize_icu`, a SQLite loadable extension that registers ICU collations under the same names PostgreSQL uses. The same locale resolves to the same collation name and produces the same ordering on either database, so a query ported between them sorts identically — the test suite asserts this for a range of languages and tailorings.
78124

79125
Install ICU (`brew install icu4c`, or `apt-get install libicu-dev`), enable the build in `config/config.exs`, and load the extension on every connection:
80126

81127
```elixir
82128
# config/config.exs — read at compile time, so not runtime.exs
83-
config :localize_ecto, :sqlite_icu, true
129+
config :localize_sql, :sqlite_icu, true
84130

85131
# config/runtime.exs
86132
config :my_app, MyApp.Repo,
87133
load_extensions: Localize.Ecto.SQLite3.Extension.load_extensions()
88134
```
89135

90-
Then import [Localize.Ecto.SQLite3](https://hexdocs.pm/localize_ecto/Localize.Ecto.SQLite3.html) in place of `Localize.Ecto`:
136+
Then import [Localize.Ecto.SQLite3](https://hexdocs.pm/localize_sql/Localize.Ecto.SQLite3.html) in place of `Localize.Ecto`:
91137

92138
```elixir
93139
import Ecto.Query
@@ -99,15 +145,15 @@ from p in Product, order_by: collate(p.name, "sv")
99145
from p in Product, order_by: collate(p.name, "de-u-co-phonebk")
100146
```
101147

102-
Collations are built the first time a query names one, so unlike PostgreSQL the BCP 47 tailorings — `-u-co-phonebk`, `-u-kn-true`, `-u-ks-level1` — need no `CREATE COLLATION` migration. The build is opt-in: without it `localize_ecto` stays a pure-Elixir package needing no C toolchain or ICU, and PostgreSQL users are unaffected.
148+
Collations are built the first time a query names one, so unlike PostgreSQL the BCP 47 tailorings — `-u-co-phonebk`, `-u-kn-true`, `-u-ks-level1` — need no `CREATE COLLATION` migration. The build is opt-in: without it `localize_sql` stays a pure-Elixir package needing no C toolchain or ICU, and PostgreSQL users are unaffected.
103149

104-
One consequence to weigh before indexing with an ICU collation: an index that names a collation makes the database unreadable by any connection that has not loaded the extension, including the `sqlite3` CLI. The [Collations in SQLite](https://hexdocs.pm/localize_ecto/collations_in_sqlite.html) guide covers this, the case mapping differences, and what does not port from PostgreSQL.
150+
One consequence to weigh before indexing with an ICU collation: an index that names a collation makes the database unreadable by any connection that has not loaded the extension, including the `sqlite3` CLI. The [Collations in SQLite](https://hexdocs.pm/localize_sql/collations_in_sqlite.html) guide covers this, the case mapping differences, and what does not port from PostgreSQL.
105151

106152
## Deterministic collations and Unicode normalization
107153

108154
On PostgreSQL, the collations this library resolves to are deterministic, which is PostgreSQL's default. SQLite has no such concept and behaves differently here — see the end of this section. A deterministic collation never treats two strings as equal unless they are byte-for-byte identical: comparison first uses the linguistic collation order, then breaks ties bytewise. This has a practical consequence for Unicode text that is not normalized. Canonically equivalent strings in different normalization forms — for example `é` as the single code point U+00E9 versus `e` followed by combining U+0301 — will sort adjacently but will never compare equal, so equality tests, `DISTINCT`, `GROUP BY`, joins on text keys, and unique indexes all see them as different values.
109155

110-
To get expected results, normalize text (NFC is the usual choice) before writing it to the database. In Elixir use [String.normalize/2](https://hexdocs.pm/elixir/String.html#normalize/2); in PostgreSQL the [normalize](https://www.postgresql.org/docs/current/functions-string.html) function and `IS NFC NORMALIZED` predicate are available for checking or repairing existing data. Alternatively, PostgreSQL supports nondeterministic collations that do compare canonically equivalent strings as equal — [Localize.Ecto.Migration.create_collation/2](https://hexdocs.pm/localize_ecto/Localize.Ecto.Migration.html#create_collation/2) can create one with `deterministic: false` — but they cannot be used with `LIKE` or pattern matching and are slower.
156+
To get expected results, normalize text (NFC is the usual choice) before writing it to the database. In Elixir use [String.normalize/2](https://hexdocs.pm/elixir/String.html#normalize/2); in PostgreSQL the [normalize](https://www.postgresql.org/docs/current/functions-string.html) function and `IS NFC NORMALIZED` predicate are available for checking or repairing existing data. Alternatively, PostgreSQL supports nondeterministic collations that do compare canonically equivalent strings as equal — [Localize.Ecto.Migration.create_collation/2](https://hexdocs.pm/localize_sql/Localize.Ecto.Migration.html#create_collation/2) can create one with `deterministic: false` — but they cannot be used with `LIKE` or pattern matching and are slower.
111157

112158
SQLite behaves as PostgreSQL's nondeterministic collations do, and it is the one place the two databases disagree. SQLite has no deterministic/nondeterministic distinction, so a comparison returns exactly what ICU says — and ICU treats canonically equivalent strings as equal. The NFC and NFD forms of `café` compare **equal** on SQLite and **unequal** on PostgreSQL. Ordering is unaffected on both, which is why sorting still agrees; it is equality, `DISTINCT`, `GROUP BY` and unique indexes that differ. Normalizing on write makes the question moot on either database, and is worth doing regardless.
113159

@@ -124,11 +170,11 @@ WHERE collprovider = 'i'
124170
ORDER BY collname;
125171
```
126172

127-
If your server's set differs materially from the snapshot, pass your own list with the `:available` option of [Localize.Ecto.Collation.collation_for/2](https://hexdocs.pm/localize_ecto/Localize.Ecto.Collation.html#collation_for/2).
173+
If your server's set differs materially from the snapshot, pass your own list with the `:available` option of [Localize.Ecto.Collation.collation_for/2](https://hexdocs.pm/localize_sql/Localize.Ecto.Collation.html#collation_for/2).
128174

129175
## Performance considerations
130176

131-
Linguistic comparison is more expensive than PostgreSQL's default byte-order comparison, and an `ORDER BY ... COLLATE` clause can only use an index that was created with the same collation. For hot queries, create an index with the collation you sort by using [Localize.Ecto.Migration.collated/2](https://hexdocs.pm/localize_ecto/Localize.Ecto.Migration.html#collated/2):
177+
Linguistic comparison is more expensive than PostgreSQL's default byte-order comparison, and an `ORDER BY ... COLLATE` clause can only use an index that was created with the same collation. For hot queries, create an index with the collation you sort by using [Localize.Ecto.Migration.collated/2](https://hexdocs.pm/localize_sql/Localize.Ecto.Migration.html#collated/2):
132178

133179
```elixir
134180
create index("products", [collated(:name, "de")])
@@ -138,14 +184,14 @@ If a PostgreSQL upgrade links a newer ICU library whose collation data changed
138184

139185
## Guides
140186

141-
* [Using Localize Ecto](https://hexdocs.pm/localize_ecto/using_localize_ecto.html) — the `collate/1,2` macros, locale resolution, and migrations.
187+
* [Using Localize SQL](https://hexdocs.pm/localize_sql/using_localize_sql.html) — the `collate/1,2` macros, locale resolution, and migrations.
142188

143-
* [Collations in PostgreSQL](https://hexdocs.pm/localize_ecto/collations_in_postgres.html) — how PostgreSQL collation works, choosing a database default, and how ICU collations relate to Localize.
189+
* [Collations in PostgreSQL](https://hexdocs.pm/localize_sql/collations_in_postgres.html) — how PostgreSQL collation works, choosing a database default, and how ICU collations relate to Localize.
144190

145-
* [Collations in SQLite](https://hexdocs.pm/localize_ecto/collations_in_sqlite.html) — the `localize_icu` extension, on-demand collation registration, and the trade-offs of indexing with an ICU collation.
191+
* [Collations in SQLite](https://hexdocs.pm/localize_sql/collations_in_sqlite.html) — the `localize_icu` extension, on-demand collation registration, and the trade-offs of indexing with an ICU collation.
146192

147193
## License
148194

149195
Copyright 2026 Kip Cole
150196

151-
Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/elixir-localize/localize_ecto/blob/v1.0.0-rc.0/LICENSE.md).
197+
Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/elixir-localize/localize_sql/blob/v1.0.0-rc.0/LICENSE.md).

0 commit comments

Comments
 (0)