Skip to content

Expand ETF universe ingestion for the technical-analysis framework (follow-up to #109) #114

Description

@C00ldudeNoonan

Context

Split out from #109 (see its acceptance criteria: "missing ETF coverage is split to a separate Dagster ingestion issue"). The technical-analysis framework in PR #113 covers all currently ingested universes:

  • sp500_companies_prices_raw (split-adjusted)
  • us_sector_etfs_raw
  • major_indices_raw
  • fixed_income_etfs_raw
  • currency_etfs_raw
  • global_markets_raw

Any ETF or symbol universe beyond these (e.g. factor ETFs beyond what exists, thematic ETFs, commodity ETFs, international sector ETFs, broader index membership) needs new MarketStack ingestion in Dagster before the technical framework can cover it.

Scope

  • Decide the target ETF universe expansion (candidates: factor ETFs referenced by Factor Analysis #52, thematic ETFs, commodity ETFs such as GLD/USO, volatility products)
  • Add a Dagster partitioned asset per new universe following the existing pattern in macro_agents/src/macro_agents/defs/domains/markets/assets.py
  • Add staging model + source entry mirroring stg_us_sectors.sql
  • Register the new staging model in technical_price_universe.sql (one-line addition to the jinja universe list)

Notes

  • Ingestion follows the existing MarketStack resource; consider the pagination/batching improvements flagged in the performance review before adding a large universe (per-partition upserts and page-size limits make large backfills slow today).

Acceptance criteria

  • New universe lands in raw + staging with the standard OHLCV shape
  • technical_price_universe includes the new universe and downstream technical models build cleanly
  • No changes to signal logic required (framework is universe-agnostic)

Metadata

Metadata

Assignees

No one assigned

    Labels

    data-pipelineDagster ingestion, scraping, or new data source workenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions