ci: reduce PR test matrix for Spark SQL and Iceberg#4404
Closed
andygrove wants to merge 1 commit into
Closed
Conversation
Pull request builds now run a reduced matrix while pushes to main and manual workflow_dispatch runs keep the full matrix: - Spark SQL tests: pull requests test only Spark 3.5 and 4.1 (down from 3.4, 3.5, 4.0, 4.1). - Iceberg tests: pull requests test only the latest Iceberg version, 1.10.0 (down from 1.8.1, 1.9.1, 1.10.0). The varying matrix dimension is selected with a conditional fromJSON expression keyed on github.event_name.
Member
Author
|
The Spark SQL changes here are replaced with #4408 |
Member
Author
|
will open new PR to reduce iceberg versions running on PRs |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #4406
Rationale for this change
The Spark SQL test workflow runs every Spark version (3.4, 3.5, 4.0, 4.1) on every pull request, and the Iceberg test workflows run every Iceberg version (1.8.1, 1.9.1, 1.10.0). This is valuable coverage on
mainbut is expensive to run on every pull request iteration. Reducing the matrix on pull requests shortens feedback time and frees runner capacity, while pushes tomainstill get the full sweep so regressions on the less-common versions are still caught before release.What changes are included in this PR?
The varying matrix dimension in each workflow is now selected with a conditional
fromJSONexpression keyed ongithub.event_name:spark_sql_test.yml: pull requests test only Spark 3.5 and 4.1; pushes tomainand manualworkflow_dispatchruns test the full set (3.4, 3.5, 4.0, 4.1). The 7-entrymodulematrix is unchanged, so pull request runs drop from 28 jobs to 14.iceberg_spark_test.yml: pull requests test only the latest Iceberg version (1.10.0); pushes tomainand manualworkflow_dispatchruns test the full set (1.8.1, 1.9.1, 1.10.0). This applies to all three Iceberg jobs (iceberg-spark,iceberg-spark-extensions,iceberg-spark-runtime).Manual
workflow_dispatchruns intentionally use the full matrix so the complete sweep can still be triggered on demand.How are these changes tested?
These are CI configuration changes. Both modified workflows pass
actionlint(the same check run byvalidate_workflows.yml), and the inline matrix JSON was verified to parse. The conditional matrix behavior will be observable on this pull request (reduced matrix) versus subsequent pushes tomain(full matrix).