Support sensors in branch deployments or provide a reliable alternative

[saksham1525]

What's the use case?

My pipeline relies on a sensor-driven ingestion process where a sensor continuously polls SQS queue messages, scrapes relevant data, and creates dynamic partitions for downstream assets.

However, since sensors aren’t supported in branch deployments, I’m unable to test the ingestion flow within branch pipelines. This makes branch-based testing redundant — I have to push all changes to the main or dev branch just to run ingestion tests.

This creates workflow friction, especially when multiple developers are testing different features in parallel, leading to repo clutter and potential conflicts.

Ideas of implementation

From what I’ve researched, this limitation exists due to resource constraints — branch deployments are mainly intended for unit testing individual assets.
That said, even if continuous sensors aren’t feasible, it would be really helpful to have a time-bound sensor or an option for sensors to manually trigger a run (given conditions for that trigger are met) within branch deployments.

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[zyd14]

I think sensor tests combined with unit tests largely cover this use case. I'd be curious to understand if there are gaps in that feature that are not allowing you to test runs, as it seems to me to be able to pretty fully test both the sensor logic and the pipeline that would be triggered given a certain cursor value.