[dagster-airlift][components 2/n] scaffolder

[dpeng817]

Summary & Motivation

Adds a components scaffolder for the airflow instance component.

In order to support multiple auth types from the same component, I'm taking auth_type as the top level parameter. However, this makes it difficult to scaffold out the actual underlying parameterization of the auth. Seems like something we might need additional tooling support for.

How I Tested These Changes

New components tests for python and yaml

[dpeng817]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [historical-runs 1/n][dagster-airlift] make monitor job range configurable #29339 : 2 dependent PRs (#29371 , #29381 )
• [dagster-airlift][components 2/n] scaffolder #29333 👈 (View in Graphite)
• [dagster-airlift][components 1/n] instance component #29331
• [dagster-airlift][print metadata 2/2] Ingest metadata from airflow logs #29313
• [dagster-airlift][print metadata 1/2] log retrieval API #29312
• Move pipes external metadata processing to its own layer. #29311
• [dagster-airlift][jobs 6/n] jobs UI #29278
• [dagster-airlift][jobs 5/n][RFC] Airflow monitoring job #29277
• [dagster-airlift][jobs 4/n] Definitions.execute_job_in_process #29243 : 1 other dependent PR (#29209 )
• [dagster-airlift][jobs 3/n] make AirflowDefinitionsData aware of jobs #29208
• [dagster-airlift][jobs 2/n] Job builder methods #29207
• [dagster-airlift][jobs 1/n] move assets def construction to common code path; other refactors #29206
• [dagster-airlift][datasets 3/3] dataset filtering #29164
• [dagster-airlift][datasets 2/3] auto-construct assets from datasets #29162
• [dagster-airlift][datasets 1/3] datasets storage layer, AirflowInstance retrieval methods #29155
• [dagster-airlift] Move around classes #29154
• [dagster-airlift] fix prop on AirflowDefinitionsData #29161
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[graphite-app]

There's an inconsistency in the auth type handling. The AirflowInstanceScaffolderParams class accepts "mwaa" as a valid auth_type, but the scaffold method doesn't implement this case and will raise a generic error. Either implement the MWAA auth type scaffolding or update the error message to specifically indicate that MWAA scaffolding is not yet implemented.

Suggested change

	if params.auth_type == "basic_auth":
	full_params["auth"] = {
	"type": "basic_auth",
	"webserver_url": '{{ env("AIRFLOW_WEBSERVER_URL") }}',
	"username": '{{ env("AIRFLOW_USERNAME") }}',
	"password": '{{ env("AIRFLOW_PASSWORD") }}',
	}
	else:
	raise ValueError(f"Unsupported auth type: {params.auth_type}")
	if params.auth_type == "basic_auth":
	full_params["auth"] = {
	"type": "basic_auth",
	"webserver_url": '{{ env("AIRFLOW_WEBSERVER_URL") }}',
	"username": '{{ env("AIRFLOW_USERNAME") }}',
	"password": '{{ env("AIRFLOW_PASSWORD") }}',
	}
	elif params.auth_type == "mwaa":
	raise NotImplementedError(
	"MWAA authentication type is recognized but scaffolding is not yet implemented"
	)
	else:
	raise ValueError(f"Unsupported auth type: {params.auth_type}")

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.

[benpankow]

yea this is not ideal but seems reasonable enough for now