Skip to content

Commit f589fdd

Browse files
authored
[dagster-dbt] Be more resilient to missing timing data from dbt cloud (#33045)
## Summary & Motivation It's possible for dbt cloud to not return any timing data in some circumstances. The main purpose of us tracking this is to be able to emit materialization events in the proper order. This gives a fallback of using the time we received the event as the completed_at time, and swaps the tuple that we're using to sort our materialization events to prioritize the topological order ## How I Tested These Changes ## Changelog [dagster-dbt] Fixed issue that could cause errors when emitting events for a dbt Cloud job run
1 parent 68bd876 commit f589fdd

2 files changed

Lines changed: 6 additions & 1 deletion

File tree

python_modules/libraries/dagster-dbt/dagster_dbt/cloud_v2/run_handler.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
get_dagster_logger,
1313
)
1414
from dagster._record import record
15+
from dagster._time import get_current_timestamp
1516
from dateutil import parser
1617
from requests.exceptions import RequestException
1718

@@ -80,6 +81,10 @@ def get_run_logs(self) -> Optional[str]:
8081

8182

8283
def get_completed_at_timestamp(result: Mapping[str, Any]) -> float:
84+
timing = result["timing"]
85+
if len(timing) == 0:
86+
# as a fallback, use the current timestamp
87+
return get_current_timestamp()
8388
# result["timing"] is a list of events in run_results.json
8489
# For successful models and passing tests,
8590
# the last item of that list includes the timing details of the execution.

python_modules/libraries/dagster-dbt/dagster_dbt/cloud_v2/sensor_builder.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -142,7 +142,7 @@ def sorted_asset_events(
142142
return [
143143
sorted_event[1]
144144
for sorted_event in sorted(
145-
materializations_and_timestamps, key=lambda x: (x[0], topo_aks.index(x[1].asset_key))
145+
materializations_and_timestamps, key=lambda x: (topo_aks.index(x[1].asset_key), x[0])
146146
)
147147
]
148148

0 commit comments

Comments
 (0)