Skip to content

[historical-runs 3/n] Make runs obey event log entry timestamp #29382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

dpeng817
Copy link
Contributor

Summary & Motivation

How I Tested These Changes

Changelog

Insert changelog entry or delete this section.

Copy link
Contributor Author

dpeng817 commented Apr 17, 2025

This stack of pull requests is managed by Graphite. Learn more about stacking.

@dpeng817 dpeng817 changed the title Make runs obey event log entry timestamp [historical-runs 3/n] Make runs obey event log entry timestamp Apr 17, 2025
@dpeng817 dpeng817 requested review from OwenKephart and prha April 17, 2025 23:29
@dpeng817 dpeng817 marked this pull request as ready for review April 17, 2025 23:29
@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from f03f254 to 90135e9 Compare April 18, 2025 17:33
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from 16ce37e to 558fac0 Compare April 18, 2025 17:33
@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from 90135e9 to 3dfbac9 Compare April 18, 2025 18:00
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from 558fac0 to 18ec729 Compare April 18, 2025 18:00
@prha prha requested review from alangenfeld and jmsanders April 22, 2025 18:14
@prha
Copy link
Member

prha commented Apr 22, 2025

Worth getting more eyes on this. This changes the run start/end to be in terms of the event timestamps rather than by the time that the instance handles the event.

@shalabhc
Copy link
Contributor

Is the update_timestamp user provided? If yes, does that mean users now control the run duration metric?

@dpeng817
Copy link
Contributor Author

dpeng817 commented Apr 22, 2025

@shalabhc the intended use case is for that to come from the event log entry - so not quite user provided but may be calculated by user code.

We'll likely fork implementations outside of OSS to maintain a cleaner user code vs db layer but this seems reasonable for OSS

@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from 3dfbac9 to 4b3b452 Compare April 23, 2025 16:52
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from 18ec729 to e2cf599 Compare April 23, 2025 16:52
@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from 4b3b452 to 103f752 Compare April 23, 2025 22:00
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from e2cf599 to dcd8450 Compare April 23, 2025 22:00
@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from 103f752 to 4503c8c Compare April 23, 2025 22:28
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from dcd8450 to ad88867 Compare April 23, 2025 22:28
@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from 4503c8c to 1a5d483 Compare April 23, 2025 23:10
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from ad88867 to 1042d1d Compare April 23, 2025 23:10
@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from 1a5d483 to 850a289 Compare April 23, 2025 23:36
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from 1042d1d to f6b9f49 Compare April 23, 2025 23:36
@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from 850a289 to 4433fb7 Compare April 24, 2025 00:08
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from f6b9f49 to 2923b33 Compare April 24, 2025 00:08
@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from 4433fb7 to 1df7391 Compare April 24, 2025 00:51
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from 813d40f to 2a1333e Compare April 24, 2025 16:54
Copy link
Member

@alangenfeld alangenfeld left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this is probably fine

Comment on lines -192 to -193
# consider changing the `handle_run_event` signature to get timestamp off of the
# EventLogEntry instead of the DagsterEvent, for consistency
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the fact this comment was already here i think helps this changes case

@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from 101bdca to 3deac73 Compare April 24, 2025 23:48
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from 2a1333e to 7e32499 Compare April 24, 2025 23:48
@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from 3deac73 to aa14550 Compare April 25, 2025 00:15
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from 7e32499 to e30a5d8 Compare April 25, 2025 00:15
@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from aa14550 to cc1adfe Compare April 25, 2025 00:30
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from e30a5d8 to bafa107 Compare April 25, 2025 00:30
@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from cc1adfe to eae751a Compare April 25, 2025 02:06
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from bafa107 to 560b46c Compare April 25, 2025 02:06
This was referenced Apr 25, 2025
@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from eae751a to 5176207 Compare April 29, 2025 00:46
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from 560b46c to a7a7759 Compare April 29, 2025 00:46
@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from 5176207 to c2c2ed2 Compare April 30, 2025 19:42
@dpeng817 dpeng817 mentioned this pull request Apr 30, 2025
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from a7a7759 to f03f68a Compare April 30, 2025 19:42
@dpeng817 dpeng817 mentioned this pull request Apr 30, 2025
@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from c2c2ed2 to fa44bf9 Compare April 30, 2025 20:47
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from f03f68a to 834b9f0 Compare April 30, 2025 20:47
)
record.created = get_current_timestamp()
record.msecs = (record.created - int(record.created)) * 1000
record.relativeCreated = record.created # this is incorrect. You really want to get the start time of the program, but we don't have a great way to do that. Since this is just for testing, we ignore the incosistency.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a typo in the comment: incosistency should be inconsistency.

Suggested change
record.relativeCreated = record.created # this is incorrect. You really want to get the start time of the program, but we don't have a great way to do that. Since this is just for testing, we ignore the incosistency.
record.relativeCreated = record.created # this is incorrect. You really want to get the start time of the program, but we don't have a great way to do that. Since this is just for testing, we ignore the inconsistency.

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.

@dpeng817 dpeng817 force-pushed the dpeng817/add_history_import_run_method branch from fa44bf9 to 602b1ed Compare May 1, 2025 00:17
@dpeng817 dpeng817 force-pushed the dpeng817/runs_obey_event_timestamp branch from 126d4fd to cbc50ed Compare May 1, 2025 00:17
@dpeng817 dpeng817 merged commit 55bf360 into dpeng817/add_history_import_run_method May 1, 2025
1 of 2 checks passed
@dpeng817 dpeng817 deleted the dpeng817/runs_obey_event_timestamp branch May 1, 2025 00:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants