-
Notifications
You must be signed in to change notification settings - Fork 1.7k
[dagster-airlift][components 2/n] scaffolder #29333
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[dagster-airlift][components 2/n] scaffolder #29333
Conversation
a091eef
to
bcdc03b
Compare
if params.auth_type == "basic_auth": | ||
full_params["auth"] = { | ||
"type": "basic_auth", | ||
"webserver_url": '{{ env("AIRFLOW_WEBSERVER_URL") }}', | ||
"username": '{{ env("AIRFLOW_USERNAME") }}', | ||
"password": '{{ env("AIRFLOW_PASSWORD") }}', | ||
} | ||
else: | ||
raise ValueError(f"Unsupported auth type: {params.auth_type}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's an inconsistency in the auth type handling. The AirflowInstanceScaffolderParams
class accepts "mwaa"
as a valid auth_type
, but the scaffold
method doesn't implement this case and will raise a generic error. Either implement the MWAA auth type scaffolding or update the error message to specifically indicate that MWAA scaffolding is not yet implemented.
if params.auth_type == "basic_auth": | |
full_params["auth"] = { | |
"type": "basic_auth", | |
"webserver_url": '{{ env("AIRFLOW_WEBSERVER_URL") }}', | |
"username": '{{ env("AIRFLOW_USERNAME") }}', | |
"password": '{{ env("AIRFLOW_PASSWORD") }}', | |
} | |
else: | |
raise ValueError(f"Unsupported auth type: {params.auth_type}") | |
if params.auth_type == "basic_auth": | |
full_params["auth"] = { | |
"type": "basic_auth", | |
"webserver_url": '{{ env("AIRFLOW_WEBSERVER_URL") }}', | |
"username": '{{ env("AIRFLOW_USERNAME") }}', | |
"password": '{{ env("AIRFLOW_PASSWORD") }}', | |
} | |
elif params.auth_type == "mwaa": | |
raise NotImplementedError( | |
"MWAA authentication type is recognized but scaffolding is not yet implemented" | |
) | |
else: | |
raise ValueError(f"Unsupported auth type: {params.auth_type}") |
Spotted by Diamond
Is this helpful? React 👍 or 👎 to let us know.
6549553
to
725ed95
Compare
bcdc03b
to
9e44fbd
Compare
725ed95
to
0adf39c
Compare
9e44fbd
to
a6dc99d
Compare
0adf39c
to
d8cf156
Compare
a6dc99d
to
8190550
Compare
if params.auth_type == "basic_auth": | ||
full_params["auth"] = { | ||
"type": "basic_auth", | ||
"webserver_url": '{{ env("AIRFLOW_WEBSERVER_URL") }}', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yea this is not ideal but seems reasonable enough for now
d8cf156
to
c35ec7a
Compare
ee04ae2
to
d650123
Compare
f94e74f
to
163b6f8
Compare
d650123
to
f046484
Compare
163b6f8
to
fdea73e
Compare
f046484
to
9db0786
Compare
fdea73e
to
020ec70
Compare
9db0786
to
99dcb50
Compare
020ec70
to
056a789
Compare
99dcb50
to
a3be2ff
Compare
056a789
to
6c3a327
Compare
a3be2ff
to
190d814
Compare
6c3a327
to
d4c19cc
Compare
190d814
to
0fb2a79
Compare
d4c19cc
to
902eb07
Compare
0fb2a79
to
b3a5542
Compare
902eb07
to
c4a0333
Compare
b3a5542
to
6bbcbf6
Compare
c4a0333
to
10d77a8
Compare
6bbcbf6
to
03fdd28
Compare
10d77a8
to
baf0d66
Compare
…rable (#29339) ## Summary & Motivation Switches the monitor job to operate over a range specified via config. This opens the door to make it usable for history import. The upshot is that we switch to a sensor instead of a schedule for kicking off compute so that we can manage the range interval we're operating over in a cursor. I think this is actually nicer / more observable overall. ## How I Tested These Changes New tests for sensor behavior, updated existing monitor job tests.
Deploy preview for dagit-core-storybook ready! ✅ Preview Built with commit d446e67. |
Summary & Motivation
Adds a components scaffolder for the airflow instance component.
In order to support multiple auth types from the same component, I'm taking auth_type as the top level parameter. However, this makes it difficult to scaffold out the actual underlying parameterization of the auth. Seems like something we might need additional tooling support for.
How I Tested These Changes
New components tests for python and yaml