Skip to content

Additional CI Checks for private-bigquery-etl / bigquery-etl #7901

@data-sync-user

Description

@data-sync-user

After discussion with :klukas I'm filing a general issue. I had originally filed bug #1759757 for a specific (private-bqetl-only) check but it's worth noting that we've had several deployment failures where manual review did not catch issues such as the following:

  1. mismatch between file name and SQL contents: https://github.com/mozilla/private-bigquery-etl/issues/84
  2. malformed sql or associated tests https://github.com/mozilla/private-bigquery-etl/issues/81 https://github.com/mozilla/private-bigquery-etl/pull/75/files
  3. incorrect ACLs for dataset_metadata.yaml https://github.com/mozilla/private-bigquery-etl/pull/80

For the SQL ones, I think there may be some amount of CI checks that exist on public bigquery-etl that aren't run in private-bigquery-etl. We should make sure to port over relevant CI checks in these cases, if any.

There are also some longstanding issues that affect public bqetl as well, such as view column mismatches: https://github.com/mozilla/private-bigquery-etl/pull/61 #2794

and making sure identifiers are fully qualified: https://github.com/mozilla/private-bigquery-etl/issues/24 #1413

Some of these are more difficult because of access restrictions for dry run service accounts, but where possible we should introduce CI checks for common issues/typos so we don't need to rely on manual review.

┆Issue is synchronized with this Jira Task

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions