Skip to content

[dagster-airlift][components 2/n] scaffolder #29333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

dpeng817
Copy link
Contributor

@dpeng817 dpeng817 commented Apr 16, 2025

Summary & Motivation

Adds a components scaffolder for the airflow instance component.

In order to support multiple auth types from the same component, I'm taking auth_type as the top level parameter. However, this makes it difficult to scaffold out the actual underlying parameterization of the auth. Seems like something we might need additional tooling support for.

How I Tested These Changes

New components tests for python and yaml

Copy link
Contributor Author

dpeng817 commented Apr 16, 2025

@dpeng817 dpeng817 requested review from OwenKephart and prha April 16, 2025 21:21
@dpeng817 dpeng817 marked this pull request as ready for review April 16, 2025 21:21
@dpeng817 dpeng817 force-pushed the dpeng817/airlift_instance_scaffolder branch from a091eef to bcdc03b Compare April 16, 2025 21:29
Comment on lines +48 to +56
if params.auth_type == "basic_auth":
full_params["auth"] = {
"type": "basic_auth",
"webserver_url": '{{ env("AIRFLOW_WEBSERVER_URL") }}',
"username": '{{ env("AIRFLOW_USERNAME") }}',
"password": '{{ env("AIRFLOW_PASSWORD") }}',
}
else:
raise ValueError(f"Unsupported auth type: {params.auth_type}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's an inconsistency in the auth type handling. The AirflowInstanceScaffolderParams class accepts "mwaa" as a valid auth_type, but the scaffold method doesn't implement this case and will raise a generic error. Either implement the MWAA auth type scaffolding or update the error message to specifically indicate that MWAA scaffolding is not yet implemented.

Suggested change
if params.auth_type == "basic_auth":
full_params["auth"] = {
"type": "basic_auth",
"webserver_url": '{{ env("AIRFLOW_WEBSERVER_URL") }}',
"username": '{{ env("AIRFLOW_USERNAME") }}',
"password": '{{ env("AIRFLOW_PASSWORD") }}',
}
else:
raise ValueError(f"Unsupported auth type: {params.auth_type}")
if params.auth_type == "basic_auth":
full_params["auth"] = {
"type": "basic_auth",
"webserver_url": '{{ env("AIRFLOW_WEBSERVER_URL") }}',
"username": '{{ env("AIRFLOW_USERNAME") }}',
"password": '{{ env("AIRFLOW_PASSWORD") }}',
}
elif params.auth_type == "mwaa":
raise NotImplementedError(
"MWAA authentication type is recognized but scaffolding is not yet implemented"
)
else:
raise ValueError(f"Unsupported auth type: {params.auth_type}")

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.

if params.auth_type == "basic_auth":
full_params["auth"] = {
"type": "basic_auth",
"webserver_url": '{{ env("AIRFLOW_WEBSERVER_URL") }}',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea this is not ideal but seems reasonable enough for now

@dpeng817 dpeng817 force-pushed the airlift_instance_component branch from d8cf156 to c35ec7a Compare April 23, 2025 16:52
@dpeng817 dpeng817 force-pushed the airlift_instance_component branch from ee04ae2 to d650123 Compare April 24, 2025 04:28
@dpeng817 dpeng817 force-pushed the dpeng817/airlift_instance_scaffolder branch from f94e74f to 163b6f8 Compare April 24, 2025 04:29
@dpeng817 dpeng817 force-pushed the airlift_instance_component branch from d650123 to f046484 Compare April 24, 2025 05:06
@dpeng817 dpeng817 force-pushed the dpeng817/airlift_instance_scaffolder branch from 163b6f8 to fdea73e Compare April 24, 2025 05:06
@dpeng817 dpeng817 force-pushed the airlift_instance_component branch from f046484 to 9db0786 Compare April 24, 2025 23:47
@dpeng817 dpeng817 force-pushed the dpeng817/airlift_instance_scaffolder branch from fdea73e to 020ec70 Compare April 24, 2025 23:47
@dpeng817 dpeng817 force-pushed the airlift_instance_component branch from 9db0786 to 99dcb50 Compare April 25, 2025 00:15
@dpeng817 dpeng817 force-pushed the dpeng817/airlift_instance_scaffolder branch from 020ec70 to 056a789 Compare April 25, 2025 00:15
@dpeng817 dpeng817 force-pushed the airlift_instance_component branch from 99dcb50 to a3be2ff Compare April 25, 2025 00:29
@dpeng817 dpeng817 force-pushed the dpeng817/airlift_instance_scaffolder branch from 056a789 to 6c3a327 Compare April 25, 2025 00:29
@dpeng817 dpeng817 force-pushed the airlift_instance_component branch from a3be2ff to 190d814 Compare April 25, 2025 02:03
@dpeng817 dpeng817 force-pushed the dpeng817/airlift_instance_scaffolder branch from 6c3a327 to d4c19cc Compare April 25, 2025 02:06
This was referenced Apr 25, 2025
@dpeng817 dpeng817 force-pushed the airlift_instance_component branch from 190d814 to 0fb2a79 Compare April 29, 2025 00:46
@dpeng817 dpeng817 force-pushed the dpeng817/airlift_instance_scaffolder branch from d4c19cc to 902eb07 Compare April 29, 2025 00:46
@dpeng817 dpeng817 force-pushed the airlift_instance_component branch from 0fb2a79 to b3a5542 Compare April 30, 2025 19:42
@dpeng817 dpeng817 force-pushed the dpeng817/airlift_instance_scaffolder branch from 902eb07 to c4a0333 Compare April 30, 2025 19:42
This was referenced Apr 30, 2025
@dpeng817 dpeng817 force-pushed the airlift_instance_component branch from b3a5542 to 6bbcbf6 Compare April 30, 2025 20:47
@dpeng817 dpeng817 force-pushed the dpeng817/airlift_instance_scaffolder branch from c4a0333 to 10d77a8 Compare April 30, 2025 20:47
@dpeng817 dpeng817 force-pushed the airlift_instance_component branch from 6bbcbf6 to 03fdd28 Compare May 1, 2025 00:17
@dpeng817 dpeng817 force-pushed the dpeng817/airlift_instance_scaffolder branch from 10d77a8 to baf0d66 Compare May 1, 2025 00:17
…rable (#29339)

## Summary & Motivation
Switches the monitor job to operate over a range specified via config.
This opens the door to make it usable for history import.

The upshot is that we switch to a sensor instead of a schedule for
kicking off compute so that we can manage the range interval we're
operating over in a cursor. I think this is actually nicer / more
observable overall.

## How I Tested These Changes
New tests for sensor behavior, updated existing monitor job tests.
@dpeng817 dpeng817 merged commit 7131af3 into airlift_instance_component May 1, 2025
1 of 2 checks passed
@dpeng817 dpeng817 deleted the dpeng817/airlift_instance_scaffolder branch May 1, 2025 00:27
Copy link

github-actions bot commented May 1, 2025

Deploy preview for dagit-core-storybook ready!

✅ Preview
https://dagit-core-storybook-7627jpi58-elementl.vercel.app
https://dpeng817-airlift-instance-scaffolder.core-storybook.dagster-docs.io

Built with commit d446e67.
This pull request is being automatically deployed with vercel-action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants