bigquery-etl CI improvements

[data-sync-user]

The bigquery-etl CI has been growing more complex and slower to run. This issue will track potential improvements and bottlenecks that were identified.

https://docs.google.com/document/d/1LEZV6hOoQPbV895r5xfiDLKkositGlRFeWUfUdRA0pQ/edit?tab=t.0#heading=h.njyob3bufn8j has a deeper investigation on each CI step

┆Issue is synchronized with this Jira Task

[data-sync-user]

➤ Anna Scholtz commented:

The longest running tasks in the generate-sql step are:

Generating derived_view_schemas and stable_views. Combined these two tasks dry run every view.sql in order to get the most recent schema. These schemas are updated so that these views have the most recent descriptions, but also the schemas are used to determine which view changed and needs to get re-deployed.
There isn’t really a straightforward way to speed these processes up much further, other than finding a different approach that doesn’t involve use needing updates to the schemas.

[data-sync-user]

➤ Anna Scholtz commented:

The tasks primarily contributing to the long run time are generate-sql(20min) → deploy-changes-to-stage(depends on what changes were made) → dry-run-sql(15min). All other tasks run in parallel and usually finish before these tasks that depend on eachother complete. So in order to bring down CI runtime, these tasks have to be optimized