Skip to content

refactor: split oversized orchestration modules and narrow broad exception handling #136

Description

@C00ldudeNoonan

Summary

The repository guideline targets 100 to 400 lines per file, but 30 production Python modules exceed 400 lines and 9 exceed 600 lines.

High-priority examples include:

  • backtest_optimizer.py at roughly 992 lines, including a very large optimization asset.
  • domain_data_fetchers.py at roughly 770 lines.
  • fomc_transcripts.py at roughly 749 lines.
  • financial_condition_index.py at roughly 695 lines.
  • calendars.py at roughly 684 lines.
  • asset_failure_sensor.py at roughly 639 lines.

There are also roughly 159 broad except Exception handlers, including silent pass and empty-result fallbacks that hide warehouse, permission, and parsing failures.

Scope

  • Split modules along cohesive domain boundaries without creating excessive tiny files.
  • Extract pure transformation and parsing functions from Dagster definitions.
  • Separate query construction, warehouse access, orchestration, and external notifications.
  • Replace broad exception handlers with expected exception types.
  • Treat optional fallbacks explicitly and record structured metadata when degradation occurs.
  • Remove unreachable exception handling and legacy compatibility names during the migration cleanup.
  • Add focused tests before each refactor.

Acceptance criteria

  • Priority modules are reduced to cohesive files near the 100 to 400 line target.
  • Long asset bodies are decomposed into independently testable functions.
  • Silent broad exception handlers are eliminated from critical paths.
  • Expected fallback behavior emits structured logs or materialization metadata.
  • Public behavior and asset keys remain stable unless a documented migration is approved.
  • Coverage does not decrease during refactoring.

Metadata

Metadata

Assignees

No one assigned

    Labels

    data-pipelineDagster ingestion, scraping, or new data source workenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions