Skip to content

21-prepared-tables.md

Latest commit

 

History

History
165 lines (130 loc) · 6.62 KB

21-prepared-tables.md

File metadata and controls

165 lines (130 loc) · 6.62 KB
Raw

21 - Prepared Tables

Prepared tables are the processor's canonical form of ActivitySim output. They hide raw file naming differences and expose stable fields for summaries and dashboard pages.

Prepare Data Flow

reader
  -> identifier and column canonicalization
  -> escort and weight normalization
  -> zone context
  -> household/person enrichment
  -> student enrollment
  -> day and vehicle preparation
  -> tour enrichment
  -> trip enrichment
  -> non-motorized distance and time-period enrichment
  -> VOT bins
  -> final casts
  -> prepared cache

The public orchestration lives in processor/prepare/enrichment/pipeline.py, with domain boundaries in processor/prepare/enrichment/domains.py.

Important Modules

Module Role
processor/prepare/reader.py Loads raw files and prepared-table inputs.
processor/prepare/enrichment/pipeline.py Public prepare entry point.
processor/prepare/enrichment/domains.py Source, people/place, mobility, and final-output orchestration.
processor/prepare/enrichment/canonicalize.py Identifier and core column normalization.
processor/prepare/enrichment/weights.py finalweight behavior.
processor/prepare/enrichment/zones.py MAZ/TAZ and geography fields.
processor/prepare/enrichment/tours.py Tour-level prepared fields.
processor/prepare/enrichment/trips.py Trip-level prepared fields.
processor/prepare/enrichment/non_motorized_distance.py Optional walk/bike distance enrichment.
processor/prepare/enrichment/time_periods.py Canonical trip and tour period fields.
processor/prepare/enrichment/finalize.py Final table casting.
processor/prepare/cache.py Prepared cache IO and manifests.

Prepared Table Names

Runtime table names are defined in processor.models.PreparedTableName:

Config/file table ID RunData/summary-contract name Meaning
households hh Households.
persons per Persons.
day day Day table when available.
tours tours Tours.
trips trips Trips.
vehicles vehicles Vehicles when available.
joint_tour_participants joint_participants Joint tour participants.
land_use land_use Land use and geography lookup data.
no file-map ID skim Optional skim_matrix support exposed as a special prepared requirement.

Use config/file IDs in files, file_map, and prepared_table_map. Use the runtime names in RunData access and @summary(required_columns=...); for example, run.per and required_columns={"per": ("person_type",)}.

Common Prepared Fields

The exact schema can differ by model and optional inputs, but summaries commonly rely on:

  • canonical IDs: household_id, person_id, tour_id, trip_id
  • purpose and mode fields: tour_purpose, trip_purpose, tour_mode, trip_mode
  • time fields: start_hour, end_hour, depart_hour
  • stop fields: num_ob_stops, num_ib_stops, num_tot_stops, stops
  • distance/geography fields: SKIMDIST, OTAZ, DTAZ, HGEO, WGEO
  • household/person aliases: HHVEH, HHSIZE, AUTOSUFF, NUMBER_HH
  • aggregation weight: finalweight

Use the prepared field when it exists rather than probing raw names in a summary or page.

This list is orientation, not a guarantee that every table has every field. For a specific summary, the generated catalog in chapter 24 is the authoritative list of required prepared columns. At runtime, @summary prerequisites and prepared-table availability metadata determine whether a calculation can run.

Inspecting An Exact Prepared Schema

There is intentionally no repository-wide dump of every column from one sample prepared cache. Raw model extensions and optional inputs make such a snapshot model-specific and quickly stale.

For the cache you are actually using:

  1. Read the run's manifest.json to find the prepared-table files and recorded availability state.
  2. Inspect the Parquet or CSV schema for the relevant table.
  3. Use processor.models.RunData names at runtime and the file/config names in Prepared Table Names.
  4. Use the generated Summary Catalog to find the exact prepared columns required by each registered summary.

Stable additions belong in the owning prepare enrichment module and should be covered by a prepare test. A row count or a column found only in one regional model output is evidence about that dataset, not part of the visualizer's portable contract.

Adding A Prepared Column

For an end-to-end worked example, see Add A Column To An Existing Prepared Table.

Use this path when many summaries/pages need the same derived field or when the field is part of canonical model-output normalization.

Checklist:

  1. Choose the owning enrichment module.
  2. Add the Polars expression or transformation in the appropriate stage.
  3. Keep missing source columns graceful when the input is optional.
  4. Add final type/cast behavior if the field must be stable.
  5. Add or update tests that prepare a minimal run and assert the new column.
  6. If a summary depends on the column, add it to that summary's contract required_columns.
  7. If a page reads it directly, add the table to required_prepared_tables.

Example pattern:

if "source_column" in state.trips.columns:
    state.trips = state.trips.with_columns(
        pl.col("source_column").cast(pl.Float64).alias("new_prepared_column")
    )

Do not add page-only formatting columns to prepared tables. Prefer page helpers or summary output columns for presentation concerns.

Using Prepared Tables As Inputs

prepared_table_map lets a config bypass raw prepare for a run:

runs:
  - label: Custom Prepared
    prepared_table_map:
      households: C:\prepared\households.parquet
      persons: C:\prepared\persons.parquet
      tours: C:\prepared\tours.parquet
      trips: C:\prepared\trips.parquet
      land_use: C:\prepared\land_use.parquet

This path assumes the supplied tables already match the prepared contract.

Adding a new prepared table type is a larger change covering config, RunData, reader, availability, cache IO, pruning, and possibly segmentation. Follow the complete worked example.

Related Chapters