-
Notifications
You must be signed in to change notification settings - Fork 124
Description
After discussion with :klukas I'm filing a general issue. I had originally filed bug #1759757 for a specific (private-bqetl-only) check but it's worth noting that we've had several deployment failures where manual review did not catch issues such as the following:
- mismatch between file name and SQL contents: https://github.com/mozilla/private-bigquery-etl/issues/84
- malformed sql or associated tests https://github.com/mozilla/private-bigquery-etl/issues/81 https://github.com/mozilla/private-bigquery-etl/pull/75/files
- incorrect ACLs for
dataset_metadata.yamlhttps://github.com/mozilla/private-bigquery-etl/pull/80
For the SQL ones, I think there may be some amount of CI checks that exist on public bigquery-etl that aren't run in private-bigquery-etl. We should make sure to port over relevant CI checks in these cases, if any.
There are also some longstanding issues that affect public bqetl as well, such as view column mismatches: https://github.com/mozilla/private-bigquery-etl/pull/61 #2794
and making sure identifiers are fully qualified: https://github.com/mozilla/private-bigquery-etl/issues/24 #1413
Some of these are more difficult because of access restrictions for dry run service accounts, but where possible we should introduce CI checks for common issues/typos so we don't need to rely on manual review.
┆Issue is synchronized with this Jira Task