[DENG-8606] Generate two tables (regular and 'use_live') for queries that run more frequently than daily #8652

curtismorales · 2025-12-28T00:20:47Z

Description

This PR adds two new flags to the bqetl query create CLI command that will generate standard ETL for queries that run more frequently than daily (and pull from live tables):
--use-live: Creates an additional _use_live table and a view that joins the two
--hourly: A special case of --use-live for queries that run hourly

There is some minor refactoring for DRY but most of the logic here is new and should not affect the existing logic.

A few notes:

This PR does not include the other portions of implementing this feature. These other pieces are almost entirely orthogonal to this one and are in separate forthcoming PRs:
1. Correctly calculating airflow dependencies between DAGs that run at different cadences
2. The logic for calculating and passing @submission_date-ish parameters based on the schedule
3. Updating managed backfills to support tables at cadences other than daily
The generated metadata for the use_live table includes the somewhat gnarly calculation of parameters that currently is used -- this will be removed in (2)
I haven't added functionality here to support specifying DAGs, so for now these flags implicitly also set the flag --no-schedule. I plan to deal with this later; I don't think it's a huge deal now that the user has to add the dag names in the metadata files
The logic in the various sql and metadata templates has gotten pretty long with this addition, so I would like to move all this templated stuff out of this function. I intend to do this in a later PR, but I would like to get this change in first
~~I'm adding a few more inline notes~~

Related Tickets & Documents

Reviewer, please follow this checklist

…daily-tables

curtismorales · 2025-12-28T00:23:44Z

bigquery_etl/cli/query.py

+    derived_path = path / project_id / dataset / table_name
+    derived_path.mkdir(parents=True)
+    if use_live:
+        use_live_table_name = name + use_live_slug + version


This results in names like my_table_v1 and my_table_use_live_v1. This is not my favorite because it means the two tables can be alphabetically separated, but it also breaks our current conventions to have anything other than the version at the end of a table name. Improving our naming standards is a larger issue that I'd prefer to not get involved in here

quibble: IMO _use_live reads very awkwardly. I'd be inclined to have the slug just be _live, or maybe _from_live.

I thought about _live but thought that could lead to confusion with the actual live tables. I like _from_live, I'll make that change

fwiw there are already some queries that follow a similar pattern and do have the _live_v<n> suffix. E.g. experiment_events_live_v1, monitoring_derived.topsites_rate_fenix_release_live_v1 and more

curtismorales · 2025-12-28T00:29:04Z

bigquery_etl/cli/query.py

+                UNION ALL
+                SELECT * FROM
+                  `{project_id}.{dataset}.{use_live_table_name}`
+                WHERE submission_date > (
+                  SELECT MAX(submission_date)
+                  FROM `{project_id}.{dataset}.{table_name}`
+                )"""


This felt to me like the best way to combine these in the view. This does mean that if the main (daily) table is missing certain days before the most recent, that data will not be filled in by the _use_live table, but that feels appropriate -- a missing day in the stable table is a genuine problem in the data; the _use_live data is ephemeral and should not be used to paper over missing historical data

curtismorales · 2025-12-31T00:38:04Z

bigquery_etl/cli/query.py

+        else:
+            macro_file.write_text(
+                reformat(
+                    f"""{{% macro {table_name}(use_live) %}}


Because we can only partition by day or by hour, there isn't really a simple standard SQL for this case. This is what I came up with, but I'm open to other suggestions

curtismorales · 2025-12-31T00:41:35Z

bigquery_etl/cli/query.py

+            parameters = [
+                "submission_hour:DATE:{{(execution_date - macros.timedelta(hours=1)).strftime('%Y-%m-%d %h:00:00')}}",
+            ]
+            destination_table = f"{use_live_table_name}${{{{(execution_date - macros.timedelta(hours=1)).strftime('%Y%m%d%h)}}}}"


These will go away once the logic for calculating these parameters is in place

note: Subtracting an hour like this isn't always necessary, especially if the hourly ETLs aren't scheduled at the top of the hour.

curtismorales · 2025-12-31T00:42:05Z

bigquery_etl/cli/query.py

+            parameters = [
+                "interval_start:DATE:{{}}",
+                "interval_end:DATE:{{(execution_date - macros.timedelta(hours=1).strftime('%Y-%m-%d %h:%m:%s'))}}",
+            ]
+            destination_table = f"{use_live_table_name}${{{{(execution_date - macros.timedelta(hours=1)).strftime('%Y%m%d)}}}}"


Same as above

scholtzan · 2026-01-02T17:55:20Z

bigquery_etl/cli/query.py

+                WHERE submission_date > (
+                  SELECT MAX(submission_date)


This could potentially create an invalid query if the source table doesn't have submission_date. I think this is fine though. CI will show a failure in the dryrun task and it's reasonable to expect the PR creator to manually adjust the query.

@curtismorales I suspect you used submission_date in these queries because submission_date was being used in the existing query.sql template, but I don't think that's a good approach to continue, as live/stable tables only contain submission_timestamp columns (submission_date columns are only present in derived tables).

I'd suggest taking this opportunity to consistently use submission_timestamp in the generated ETL queries and as the default partitioning column (i.e. presuming the ETL will pass through submission_timestamp, which I think is a safer bet than presuming the ETL is going to have a derived submission_date column).

scholtzan · 2026-01-02T18:03:16Z

bigquery_etl/cli/query.py

+                + "\n"
+            )
+        click.echo(f"Created base query in {macro_file}")
+        query_file = derived_path / "query.sql"


might be worth creating a QUERY_FILE = "query.sql" here:

bigquery-etl/bigquery_etl/cli/query.py

Line 87 in 88cddaa

MATERIALIZED_VIEW = "materialized_view.sql"

since there are multiple usages.
Similar to what is done for materialized_view.sql etc

scholtzan · 2026-01-02T18:05:17Z

bigquery_etl/cli/query.py

+                expiration_days=30,
+            )
+            parameters = [
+                "interval_start:DATE:{{}}",


Should interval_start be set to something for now?

sean-rose · 2026-01-06T00:42:06Z

bigquery_etl/cli/query.py

+@click.option(
+    "--use_live",
+    "--use-live",
+    help=(
+        """Using this option creates a query that consists of two tables with
+        different schedules based on a single base query, one that runs daily
+        and pulls from stable tables and another that runs more frequently and
+        pulls from live tables, plus a view that unions the two tables.
+        """
+    ),
+    default=False,
+    is_flag=True,
+)


quibble: While the description explains the nuance/meaning of this --use-live option, IMO the option name by itself implies "use live rather than stable".

I'd be inclined to name the option something like --use-live-and-stable or --from-live-and-stable.

sean-rose · 2026-01-06T00:42:14Z

bigquery_etl/cli/query.py

+@click.option(
+    "--hourly",
+    help=(
+        """This options is a special case of the --use-live option for
+        tables that update hourly.
+
+        Using this option creates a query that consists of two tables with
+        different schedules based on a single base query, one that runs daily
+        and pulls from stable tables and another that runs hourly and
+        pulls from live tables, plus a view that unions the two tables.
+        """
+    ),
+    default=False,
+    is_flag=True,
+)


issues (blocking):

While the description explains the nuance/meaning of this --hourly option, IMO it's unnecessarily confusing, as just based on the name I think people could reasonably expect to be able to run bqetl query create --hourly <name> and get an hourly ETL that doesn't necessarily involve a combination of live and stable table sources.

This --hourly option is currently making the live ETL table use hourly partitioning, but to date hourly partitioning has been used very rarely (I currently only see one ETL using hourly partitioning), and I don't think it'd be advisable to encourage hourly partitioning due to some drawbacks:

It precludes the ETL doing aggregation at the daily level, which is a very common use case.

It makes it critical to ensure the success of all hourly ETL tasks to avoid missing any data, which is problematic because frequently running ETLs are more likely have failures (e.g. due to transient issues with BigQuery or GKE), and such ETL failures are prone to getting missed during Airflow triage as by the time triage happens there could easily have been subsequent successful hourly DAG runs.

I'd be inclined to remove this --hourly option and have the generated live ETL simply assume an hourly schedule with daily partitioning, as that's the most common sub-daily ETL setup being used thus far.

sean-rose · 2026-01-06T00:44:08Z

bigquery_etl/cli/query.py

+        no_schedule = True
+    elif use_live:
+        use_live_slug = "_use_live"
+        no_schedule = True


suggestion (non-blocking): I think it would make sense to respect the --dag argument if it's specified (for the main ETL).

sean-rose · 2026-01-06T02:10:12Z

bigquery_etl/cli/query.py

+                AS SELECT * FROM
+                  `{project_id}.{dataset}.{table_name}`"""
+
+        view_file.write_text(reformat(view_text) + "\n")


suggestion (non-blocking): I know this approach of manually appending a newline is just following the convention of what the existing code was already doing, but it's worth noting that reformat() accepts a trailing_newline boolean argument specifically for that purpose.

Suggested change

view_file.write_text(reformat(view_text) + "\n")

view_file.write_text(reformat(view_text, trailing_newline=True))

(ditto for the other similar usages)

sean-rose · 2026-01-06T02:23:39Z

bigquery_etl/cli/query.py

+    if use_live:
+        use_live_metadata_file = use_live_path / "metadata.yaml"
+        if hourly:
+            labels = {"incremental": True, "schedule": hourly}


suggestion (blocking):

Suggested change

labels = {"incremental": True, "schedule": hourly}

labels = {"incremental": True, "schedule": "hourly"}

sean-rose · 2026-01-06T02:53:06Z

bigquery_etl/cli/query.py

+                "interval_start:DATE:{{}}",
+                "interval_end:DATE:{{(execution_date - macros.timedelta(hours=1).strftime('%Y-%m-%d %h:%m:%s'))}}",


suggestion (blocking):

Suggested change

"interval_start:DATE:{{}}",

"interval_end:DATE:{{(execution_date - macros.timedelta(hours=1).strftime('%Y-%m-%d %h:%m:%s'))}}",

"interval_start:TIMESTAMP:{{}}",

"interval_end:TIMESTAMP:{{(execution_date - macros.timedelta(hours=1).strftime('%Y-%m-%d %H:%M:%S'))}}",

Though I'm not sure subtracting an extra hour would necessarily always be advisable for arbitrary intervals.

sean-rose · 2026-01-06T02:59:24Z

bigquery_etl/cli/query.py

+                "date_partition_parameter": None,
+                "parameters": parameters,
+                "destination_table": destination_table,
+                "query_file_path": use_live_path / "query.sql",


suggestion (blocking): Specifying query_file_path like this isn't actually necessary and makes maintenance harder.

Suggested change

"query_file_path": use_live_path / "query.sql",

sean-rose · 2026-01-06T05:04:33Z

bigquery_etl/cli/query.py

+            parameters = [
+                "submission_hour:DATE:{{(execution_date - macros.timedelta(hours=1)).strftime('%Y-%m-%d %h:00:00')}}",
+            ]
+            destination_table = f"{use_live_table_name}${{{{(execution_date - macros.timedelta(hours=1)).strftime('%Y%m%d%h)}}}}"


suggestion (blocking):

Suggested change

parameters = [

"submission_hour:DATE:{{(execution_date - macros.timedelta(hours=1)).strftime('%Y-%m-%d %h:00:00')}}",

]

destination_table = f"{use_live_table_name}${{{{(execution_date - macros.timedelta(hours=1)).strftime('%Y%m%d%h)}}}}"

parameters = [

"submission_hour:TIMESTAMP:{{(execution_date - macros.timedelta(hours=1)).strftime('%Y-%m-%d %H:00:00')}}",

]

destination_table = f"{use_live_table_name}${{{{(execution_date - macros.timedelta(hours=1)).strftime('%Y%m%d%H)}}}}"

sean-rose · 2026-01-06T05:05:29Z

bigquery_etl/cli/query.py

+                        WHERE
+                            {{% if use_live %}}
+                            submission_timestamp >= @interval_start
+                            AND submission_timestamp < @interval_end
+                            {{% else %}}
+                            TIMESTAMP_TRUNC(submission_date, DAY) = @submission_date
+                            {{% endif %}}
+                        {{% if use_live %}}
+                        -- Overwrite the daily partition with a combination of new records for
+                        -- the given interval (above) and existing records outside the given
+                        -- interval (below)
+                        UNION ALL
+                        SELECT
+                            *
+                        FROM
+                            {project_id}.{dataset}.{use_live_table_name}
+                        WHERE
+                            TIMESTAMP_TRUNC(submission_date, DAY) = TIMESTAMP_TRUNC(@interval_start, DAY)
+                            AND (
+                                submission_timestamp < @interval_start
+                                OR submission_timestamp >= @interval_end
+                            )
+                        {{% endif %}}


issue: submission_date wouldn't be a timestamp.

suggestion (non-blocking): It would be best to quote the live ETL table name.

note: WhiIe this approach of only running the live ETL on new records and reusing existing records is likely better for performance, it unfortunately has the same drawbacks as when using hourly partitioning that I mentioned in my comment about the --hourly command line option. I'd be inclined to run the live ETL on all records for the date in question.

sean-rose · 2026-01-06T06:10:05Z

bigquery_etl/cli/query.py

+                        FROM
+                            {{% if use_live %}}
+                            table_live


suggestion (non-blocking): IMO it would be best for the generated SQL for selecting from the live table to attempt to deduplicate the records for that date.

sean-rose · 2026-01-06T06:47:33Z

bigquery_etl/cli/query.py

+                            {{% if use_live %}}
+                            table_live
+                            {{% else %}}
+                            table_stable
+                            {{% endif %}}


quibble: I think it'd be more readable to have these placeholders match how we refer to live/stable tables.

Suggested change

{{% if use_live %}}

table_live

{{% else %}}

table_stable

{{% endif %}}

{{% if use_live %}}

live_table

{{% else %}}

stable_table

{{% endif %}}

curtismorales added 11 commits November 20, 2025 20:29

Add sub-daily option to bqetl create

6d8e0cd

Change option to sub_daily

cdeab6c

Refactor logic for creating new query directories

9d54b7e

Merge branch 'main' into DENG-8806-generate-multiple-queries-for-sub-…

8b05ccb

…daily-tables

Add --sub-daily option

835852e

Options to generate sub-daily and hourly tables

d80018e

Change name to use_live

88965cd

Merge branch 'main' into DENG-8806-generate-multiple-queries-for-sub-…

df881ce

…daily-tables

Fix typo

e4f5f65

Quotes around macro referenced by filename

d88a103

Update log statements

88cddaa

curtismorales requested a review from a team as a code owner December 28, 2025 00:20

curtismorales commented Dec 31, 2025

View reviewed changes

scholtzan reviewed Jan 2, 2026

View reviewed changes

sean-rose reviewed Jan 6, 2026

View reviewed changes

	view_file.write_text(reformat(view_text) + "\n")
	view_file.write_text(reformat(view_text, trailing_newline=True))

	labels = {"incremental": True, "schedule": hourly}
	labels = {"incremental": True, "schedule": "hourly"}

		"interval_start:DATE:{{}}",
		"interval_end:DATE:{{(execution_date - macros.timedelta(hours=1).strftime('%Y-%m-%d %h:%m:%s'))}}",

[DENG-8606] Generate two tables (regular and 'use_live') for queries that run more frequently than daily #8652

Are you sure you want to change the base?

[DENG-8606] Generate two tables (regular and 'use_live') for queries that run more frequently than daily #8652

Conversation

curtismorales commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Tickets & Documents

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

curtismorales commented Dec 28, 2025 •

edited

Loading