[historical-runs 3/n] Make runs obey event log entry timestamp

[dpeng817]

Summary & Motivation

How I Tested These Changes

Changelog

Insert changelog entry or delete this section.

[dpeng817]

• [historical-runs 3/n] Make runs obey event log entry timestamp #29382 : 2 dependent PRs (#29406 , #29594 ) 👈 (View in Graphite)
• [historical-runs 2/n] Add history import run method #29381
• [historical-runs 1/n][dagster-airlift] make monitor job range configurable #29339 : 1 other dependent PR (#29371 )
• [dagster-airlift][components 2/n] scaffolder #29333
• [dagster-airlift][components 1/n] instance component #29331
• [dagster-airlift][print metadata 2/2] Ingest metadata from airflow logs #29313
• [dagster-airlift][print metadata 1/2] log retrieval API #29312
• Move pipes external metadata processing to its own layer. #29311
• [dagster-airlift][jobs 6/n] jobs UI #29278
• [dagster-airlift][jobs 5/n][RFC] Airflow monitoring job #29277
• [dagster-airlift][jobs 4/n] Definitions.execute_job_in_process #29243 : 1 other dependent PR (#29209 )
• [dagster-airlift][jobs 3/n] make AirflowDefinitionsData aware of jobs #29208
• [dagster-airlift][jobs 2/n] Job builder methods #29207
• [dagster-airlift][jobs 1/n] move assets def construction to common code path; other refactors #29206
• [dagster-airlift][datasets 3/3] dataset filtering #29164
• [dagster-airlift][datasets 2/3] auto-construct assets from datasets #29162
• [dagster-airlift][datasets 1/3] datasets storage layer, AirflowInstance retrieval methods #29155
• [dagster-airlift] Move around classes #29154
• [dagster-airlift] fix prop on AirflowDefinitionsData #29161 : 1 other dependent PR (#29706 )
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[prha]

Worth getting more eyes on this. This changes the run start/end to be in terms of the event timestamps rather than by the time that the instance handles the event.

[shalabhc]

Is the update_timestamp user provided? If yes, does that mean users now control the run duration metric?

[dpeng817]

@shalabhc the intended use case is for that to come from the event log entry - so not quite user provided but may be calculated by user code.

We'll likely fork implementations outside of OSS to maintain a cleaner user code vs db layer but this seems reasonable for OSS

[alangenfeld]

i think this is probably fine

[alangenfeld]

the fact this comment was already here i think helps this changes case

[graphite-app]

There's a typo in the comment: incosistency should be inconsistency.

Suggested change

	record.relativeCreated = record.created # this is incorrect. You really want to get the start time of the program, but we don't have a great way to do that. Since this is just for testing, we ignore the incosistency.
	record.relativeCreated = record.created # this is incorrect. You really want to get the start time of the program, but we don't have a great way to do that. Since this is just for testing, we ignore the inconsistency.

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.