A few improvements to dagster

[bryan-harter]

Change Summary

Overview

These are a few small unrelated changes, but related to the discussions we had about dagster the other day:

1. The default sensor time is now 5 minutes instead of 15. It was largely 15 just to stop the system from being overwhelmed by the initial processing efforts when dagster turned on for the first time, but 5 minutes should be plenty now

2. Removed the code the code that prevented failed jobs from running again with no changes. This again was something to help prevent a boatload of processing runs from occurring during the early deployment steps, but now this is causing a hassle and should just be better to re-run.

3. Increased the allowed batch job timeout to 5 hours. Leaving it at 1 hour caused an issues where the job completed successfully, but Dagster didn't know about it.

4. Perhaps the nicest one - instead of skipping an asset when we don't have the dependencies, instead we return an "AssetObservation". This keeps the little icon "grey", but it actually returns metadata about the asset at that partition. Now you can click on a grey partition for an asset, and the metadata tells you exactly what dependency it failed to retrieve.

File changes

sds_data_manager/orchestration/config.py

• Setting a 5 minute sensor timer instead of 15

sds_data_manager/orchestration/imap_job.py

• 5 hour timeout
• Added AssetObservation
• Removed "Already Failed" batch job checking

Testing

tests/orchestration/test_spacecraft.py

• Fixed a unit test to verify that an AssetObservation is returned instead of nothing if we don't meet the dependencies.

[Copilot]

Pull request overview

This PR makes several operational adjustments to the Dagster orchestration layer in sds_data_manager, aiming to increase scheduling responsiveness, improve rerun ergonomics, and provide better visibility when partitions can’t run due to missing dependencies.

Changes:

• Reduce default sensor polling intervals from 15 minutes to 5 minutes across data levels.
• Increase Batch job timeout to 5 hours and remove the “already failed with same deps” resubmission guard.
• Emit AssetObservation metadata when dependencies are missing, and update tests accordingly.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
sds_data_manager/orchestration/config.py	Updates sensor schedule defaults to run more frequently (5 minutes).
sds_data_manager/orchestration/imap_job.py	Increases Batch timeout, removes “already failed” short-circuit, and adds missing-dependency AssetObservation emissions.
tests/orchestration/test_spacecraft.py	Updates the missing-dependency test expectations to reflect AssetObservation behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[tmplummer]

Nice improvements! Interested in responses to the Copilot comments but otherwise, looks good to me.

[tech3371]

Thank you for quick updates!