Skip to content

NTD: resolve competing mart tables and schemas for NTD data #3848

Open
@charlie-costanzo

Description

@charlie-costanzo

Currently, we unintentionally have two sets of mart tables focusing on NTD ridership:

  • One is currently used for analysis (these are found in warehouse/models/mart/ntd, although this schema name is not ideal)
    • A lot of work has been done to make these mature and useful, we want to preserve this logic and structure
  • One was created alongside the other NTD data domains while we were ingesting all of the NTD endpoints in bulk (these are found in warehouse/models/mart/ntd_ridership)
    • These tables pretty much just expose the staging tables as mart (very very simple, almost 1:1 with staging tables, probably shouldn't be used anywhere and don't need to exist, but have the preferable schema name of mart/ntd_ridership)

We would like to keep the tables that are found in the first location (mart/ntd), but locate them in the schema of the second location (mart/ntd_ridership –  a much preferred schema name)

To do this, we will:

  • Copy the mart/ntd tables to mart/ntd_ridership
  • Make the old tables found in mart/ntd just views of the new ones in mart/ntd_ridership
  • Add a deprecation message on the old tables
  • Then share in analyst channels in slack and give analysts time to replace the old schema name with the new schema for the tables they have been using, probably ping Christian about this directly as well

We will also need to

  • Enrich the tables in mart/ntd with caltrans_district (this has so far only been done in the mart/ntd_ridership tables)
  • Go back and remove table materializations of ntd_ridership staging models that was done in ntd: enrich tables with caltrans_district and slight refactoring, casting #3847 (this was only done temporarily while both mart schemas were referencing the same staging tables)
    • stg_ntd__complete_monthly_ridership_with_adjustments_and_estimates__upt
    • stg_ntd__complete_monthly_ridership_with_adjustments_and_estimates__voms
    • stg_ntd__complete_monthly_ridership_with_adjustments_and_estimates__vrh
    • stg_ntd__complete_monthly_ridership_with_adjustments_and_estimates__vrm

Metadata

Metadata

Labels

data-pipeline-ingestion-and-modelingIngesting, parsing and modeling data. Evan Siroky is product owner.product: ntd-ingestionIssues related to ingesting NTD data products. Product owner is Vivek.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions