Robust ignore patterns for `code.zip` in pyspark step launcher

[NazyS]

What's the use case?

It would be greate to have a more robust excluding patterns

dagster/python_modules/libraries/dagster-pyspark/dagster_pyspark/utils.py

Line 7 in 55123a9

DEFAULT_EXCLUDE = [

used for preparing code.zip for

dagster/python_modules/libraries/dagster-databricks/dagster_databricks/databricks_pyspark_step_launcher.py

Line 192 in 55123a9

class DatabricksPySparkStepLauncher(StepLauncher):

Additional arg for step launcher which could be provided by this PR #18151 would still be tedious to use and would require constant updates to codebase during development.

As an idea, Dagster could use usually already existing .gitignore for patterns and additional allow/block lists as arguments for step launcher

Ideas of implementation

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.