tracking: integrate MDIMs and charts

[lucasrodes]

One-liner

Integrate MDIM and Chart creation in ETL under the same space.

Context

Creating a chart or an MDIM should feel the same. A chart is an MDIM with a single dimension and view. The two should share the same creation path.

Currently, MDIMs live in ETL as steps export://multidim/, with their processing logic in the etl.collection module. This was a good start, but we can structure this better as we bring charts to this space.

Lines of work

Our vision is to move this to a new space and define the charts there as well. Some ideas/steps:

• Charts in ETL. At the moment, charts can only be defined by the Admin. Let's allow them to be defined in the ETL as single-dimensional MDIMs. That is, using the existing logic from etl.collection.
• New charts:// channel. Move MDIM/chart steps out of export:// into a dedicated charts:// channel. Rationale: export:// steps depending on export:// steps is conceptually muddled, and charts are central enough to OWID to deserve their own namespace.
• Terminology: prefer "Chart" over "MDIM" in user-facing surfaces. "MDIM" stays as an internal term where useful, but docs, CLI output, and skill names should default to "Chart".
• Workflow to create MDIMs/Charts in ETL. Let's make sure these remain evergreen and possibly improve them!
  • VS Code extension: We can now preview charts from VS Code. Let's improve this space, also allow for MDIM preview, etc.
  • Claude skill: .claude/skills/create-multidim/SKILL.md

Open questions

• Migration path for existing export://multidim/... steps: alias, deprecate-and-warn, or hard rename with a coordinated PR?
• Do we keep the etl.collection module name or rename it alongside the channel?
• Backward compatibility for catalog consumers referencing the old paths?

Tasks

List here tasks/PRs as we work on them

• 🎉 Charts as first-class ETL citizens via simple mdims #5969

[Marigold]

There's also a skill .claude/skills/create-multidim/SKILL.md we should check.

[lucasrodes]

@Marigold good point. I've added a point in the workflow that addresses that. Please feel free to amend the issue description.

[pabloarosado]

Thanks for creating this issue, Lucas! I agree with all your points.
I'm wondering what would be the "channel" of charts: export://multidim/ -> chart://???/
Maybe this could be a good place to start distinguishing different types of visualizations, e.g.:

• static,
• insights (specific views of charts, but not updateable),
• narrative (specific views of charts, but updateable),
• evergreen (or whatever we want to call normal updateable charts).

I haven't thought much about it, just sharing ideas.

[lucasrodes]

@pabloarosado, nice thoughts! I've added this point to the agenda for next Monday!

I'm wondering what would be the "channel" of charts: export://multidim/ -> chart://???/

I initially thought of just jumping directly to the namespace, so we'd have charts://un/latest/un_wpp. But I can totally see the motivation for differentiating "chart types". Also, it adds a degree of flexibility to the catalog path, in case things change in the future and we need to adjust the naming.

Other half-baked thoughts

I guess we could migrate:

• export://static_viz → charts://static/ (although I think that the etl output for static viz is remixed later in Figma. Maybe similar to what happens with charts in DB?)
• export://multidim/ → charts://evergreen (I'm not sold on evergreen, but we'll figure out a better name)

And then create new insights and narrative channels.

• insights I find somehow similar to static, as they are also static & remixed in Figma. They typically come from narrative charts also.
• narrative I initially thought of them as "children" of evergreen, but I think that is not necessarily the case, as they can diverge.

I actually wonder whether we would want to differentiate charts (whether dimensionless or with dimensions) from "explorers" (or whatever we end up calling them; here I'm thinking of a collection of indicators). I'm not thinking about differentiating infrastructure; just with a namespace so one can quickly see this is a simple chart, this is a collection of charts, etc.

[lucasrodes]

Plan: introduce a chart:// prefix with two symmetric channels — chart://chart/{ns}/{v}/{short} for charts (1-view or N-view, what we today call MDIMs) and chart://explorer/{ns}/{v}/{short} for explorer-style collections. Use chart as the user-facing internal term (retire multidim from path/CLI surfaces) and rename etl.collection → etl.charts alongside.

Why this shape

• Symmetric over asymmetric. A flat-vs-marked layout (chart://{ns}/... for charts, chart://explorer/{ns}/... for explorers) was tempting to drop the chart/chart redundancy, but it forces parser special-casing in parse_step / PathFinder / catalog regex. Keeping prefix://channel/ns/version/short_name uniform is simpler than saving four characters in the path.
• Two channels, not three. This issue itself says a chart is a 1-view MDIM, so charts and MDIMs share the same chart channel — view count is a config detail, not a path-level commitment that has to be renamed when a chart grows a dimension.
• explorer earns its own channel because it's a real product/infra distinction today (different UX, tolerate_extra_indicators=True, indicator heterogeneity). During tracking: Migrate Explorers into MDIMs #6014's transition we want grep -r 'chart://explorer/' dag/ to keep working as a tracking story. The channel can be dropped (or kept indefinitely as a UX marker) once feature parity lands.
• "Chart" replaces "multidim" as the surface term — multidim reads as jargon. "MDIM" stays internal where useful.
• Pablo's static / narrative / insights / evergreen axis is cross-cutting usage metadata (any chart can be any of those), so it belongs in the YAML config — not in the catalog path, where it would conflate two orthogonal axes.

Migration outline

1. Add chart:// prefix; alias export://multidim/... and export://explorers/... for one release cycle.
2. Move on disk:
   • etl/steps/export/multidim/{ns}/{v}/{short}.py → etl/steps/chart/chart/{ns}/{v}/{short}.py
   • etl/steps/export/explorers/{ns}/{v}/{short}.py → etl/steps/chart/explorer/{ns}/{v}/{short}.py (singular explorer)
3. Rename etl.collection → etl.charts; update imports across the repo.
4. Touch points: etl/steps/__init__.py (parse_step), etl/paths.py, lib/catalog/owid/catalog/core/paths.py, etl/helpers.py (PathFinder regex), etl/command.py, dag yamls, .claude/skills/create-multidim/ → create-chart.
5. Drop the alias the release after. Defer the explorer→chart fold to tracking: Migrate Explorers into MDIMs #6014 track 2.

Open questions:

• Alias-with-deprecation, or hard rename in a coordinated PR?
• After tracking: Migrate Explorers into MDIMs #6014 track 2, drop the explorer channel, or keep it indefinitely as a UX marker?