Skip to content

feat: add commodity ETF ingestion universe#123

Merged
C00ldudeNoonan merged 1 commit into
mainfrom
codex/issue-114-commodity-etfs
Jul 5, 2026
Merged

feat: add commodity ETF ingestion universe#123
C00ldudeNoonan merged 1 commit into
mainfrom
codex/issue-114-commodity-etfs

Conversation

@C00ldudeNoonan

Copy link
Copy Markdown
Owner

Summary

  • Fixes Expand ETF universe ingestion for the technical-analysis framework (follow-up to #109) #114
  • Adds a 12-symbol commodity ETF MarketStack universe: GLD, SLV, USO, UNG, DBA, DBC, CORN, WEAT, SOYB, CPER, PPLT, PALL.
  • Registers commodity_etfs_raw in Dagster partitions, asset, job, checks, and definitions.
  • Adds stg_commodity_etfs plus source/docs/tests and includes commodity_etf in technical_price_universe.
  • Includes commodity_etfs_raw in current_data_coverage and OHLC data-quality routing.
  • Fixes shared weekly coverage asset-check SQL to use BigQuery-native date generation/date truncation.

Validation

  • uv run ruff check .
  • uv run ruff format --check .
  • uv run --with duckdb pytest tests/test_schedules.py tests/test_market_checks.py -q
  • uv run dg check defs
  • uv run --project macro_agents dbt parse --project-dir dbt_project --profiles-dir dbt_project
  • uv run --project macro_agents dbt ls --project-dir dbt_project --profiles-dir dbt_project --select stg_commodity_etfs+
  • uv run --project macro_agents dbt compile --project-dir dbt_project --profiles-dir dbt_project --select stg_commodity_etfs technical_price_universe current_data_coverage
  • uv run dg launch --job commodity_etfs_ingestion_job --partition "2026-07-01|GLD"
  • uv run --project macro_agents dbt build --project-dir dbt_project --profiles-dir dbt_project --select stg_commodity_etfs technical_price_universe current_data_coverage

Notes

  • Smoke backfill loaded only GLD for the current month; the commodity ETF coverage check executes now but warns until the full universe/history is backfilled.

@C00ldudeNoonan

Copy link
Copy Markdown
Owner Author

@codex please review this PR.

@C00ldudeNoonan
C00ldudeNoonan merged commit 8d4c3de into main Jul 5, 2026
2 checks passed
@C00ldudeNoonan
C00ldudeNoonan deleted the codex/issue-114-commodity-etfs branch July 5, 2026 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expand ETF universe ingestion for the technical-analysis framework (follow-up to #109)

1 participant