Skip to content

Clear the column-description divergence backlog, then promote column-description-divergent to HARD #532

Description

@cboettig

Follow-on to #512, which added the detector but deliberately shipped it ADVISORY.

The defect class

get_stac_details folds per-column descriptions across every asset in a collection and first-seen wins. A column documented two ways therefore loses one version silently — and whichever asset happens to be listed first in the JSON decides what every consumer reads about that column on all assets.

Two ways it bites, both found live on ca30x30-conserved-areas-terrestrial-2025 (#512):

  • A hex-only clause is dropped. Eight columns appended **…repeated on every hex cell — do not SUM on hex; dedup by _cng_fid first** to text that also existed on the flat GeoParquet. The flat is listed first, so those eight duplication warnings were invisible to every consumer for as long as they existed. That is the whole point of the warning, silently defeated.
  • A stale text outlives its asset and becomes false. Adding the hex-weights assets with their own h10 wording lost to the older hex asset, whose text reads "one row per (feature, h10) pair" — true there, wrong for the per-cell assets, which are one row per cell. The rendered schema actively misdescribed the new assets.

Neither is visible from reading any single asset. Both were found only by diffing the rendered get_stac_details output against the STAC.

Detector

verify-stac.py check_column_description_consistency (shipped in #512) reports column-description-divergent, naming the divergent assets and the winning text's asset. Unit-tested in #531.

It is ADVISORY, not HARD, purely because there is a pre-existing backlog and hard-failing would block unrelated PRs on debt they did not create.

The backlog

Spot-checked, not exhaustive:

Collection divergent columns
wdpa 17
cwhr13 8
census-2024/tract 2
padus-4-1/fee 1
ca30x30-conserved-areas-terrestrial-2025 0 (fixed in #512)
ca30x30-ecoregion 0 (fixed in #512)

A full count needs the catalog-wide pass in #509; treat these as evidence the backlog is real and modest, not as its size.

Done when

  1. Every collection reports zero column-description-divergent.
  2. check_column_description_consistency is promoted from ADVISORY to HARD (a one-word change plus the docstring rationale), so the silent drop cannot recur.
  3. A test asserting the HARD severity is added alongside the existing ones in tests/test_verify_stac.py.

How to fix a divergence

Resolve to one text per column name, and put anything asset-specific in that asset's own description, which the fold always renders:

  • Prefer the flat GeoParquet text as canonical (per AGENTS.md), merging in anything the other variant said that it did not — do not just delete the loser, or you drop real content, which is how the eight warnings vanished in the first place.
  • Hex-specific duplication notes belong in the hex asset description, not per column.
  • Keep shared H3 index columns grain-neutral, since the same column name appears on assets with different row grains.

Worked example: #512's conserved-areas pass reconciled all 32 columns and relocated the duplication warning to a new hex asset description.

Note on lean PMTiles

The detector skips columns with no description, so a lean PMTiles asset (name + type + values, prose omitted per AGENTS.md) never trips it — omitting text is not disagreeing about it. Making a non-lean PMTiles asset lean is therefore one way to clear its divergences, but not required, and it costs nothing in styling: values is retained precisely because it is what MapLibre filtering and colouring need.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions