[dagster-airlift][datasets 3/3] dataset filtering

[dpeng817]

Summary & Motivation

Adds filtering properties for datasets.

• completely turn off retrieval of datasets
• only retrieve dataset information for producing dags that match the rest of the filtering.
• allow filtering on uri pattern.

How I Tested These Changes

Additional tests.

Changelog

• [dagster-airlift] The AirflowFilter API has been updated with options to limit the retrieval of datasets from the rest API.

[dpeng817]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [dagster-airlift][jobs 4/n] Definitions.execute_job_in_process #29243 : 2 dependent PRs (#29209 , #29277 )
• [dagster-airlift][jobs 3/n] make AirflowDefinitionsData aware of jobs #29208
• [dagster-airlift][jobs 2/n] Job builder methods #29207
• [dagster-airlift][jobs 1/n] move assets def construction to common code path; other refactors #29206
• [dagster-airlift][datasets 3/3] dataset filtering #29164 👈 (View in Graphite)
• [dagster-airlift][datasets 2/3] auto-construct assets from datasets #29162
• [dagster-airlift][datasets 1/3] datasets storage layer, AirflowInstance retrieval methods #29155
• [dagster-airlift] Move around classes #29154
• [dagster-airlift] fix prop on AirflowDefinitionsData #29161
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[graphite-app]

The current implementation checks for substring presence with retrieval_filter.dataset_uri_ilike not in dataset.uri, which doesn't accurately simulate SQL's ILIKE operator behavior. For better fidelity with the real implementation, consider using case-insensitive pattern matching (e.g., with regex) or at minimum a case-insensitive substring check. This would more closely match how ILIKE works in the actual SQL queries used by the production code.

Suggested change

	if (
	retrieval_filter.dataset_uri_ilike
	and retrieval_filter.dataset_uri_ilike not in dataset.uri
	):
	if (
	retrieval_filter.dataset_uri_ilike
	and retrieval_filter.dataset_uri_ilike.lower() not in dataset.uri.lower()
	):

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.

[OwenKephart]

would prefer to match the python name to the REST API name, e.g. just rename dataset_uri_ilike to dataset_uri_pattern.