Skip to content

Create job backfilling script#1280

Merged
jjnesbitt merged 4 commits intomainfrom
backfill-jobs-command
Dec 22, 2025
Merged

Create job backfilling script#1280
jjnesbitt merged 4 commits intomainfrom
backfill-jobs-command

Conversation

@jjnesbitt
Copy link
Collaborator

Adapted from @mvandenburgh's initial work.

There was a previous iteration that used the web_hook_logs table. As far as I can tell, that table has been done away with, in favor of a web_hook_logs_daily table, which is somewhat ephemeral. So, we are forced to resort back to constructing the webhook from the underlying db data.

There are other miscellaneous fixes included, which would prevent the backfill script from properly running. These include adding the dotenv job type, and using a different yaml loader (yaml.safe_load is very slow).

@jjnesbitt jjnesbitt force-pushed the backfill-jobs-command branch from e1778c0 to 0fd1bd5 Compare December 22, 2025 19:56
The `yaml.safe_load` function is much slower
@jjnesbitt jjnesbitt force-pushed the backfill-jobs-command branch 3 times, most recently from 12c2650 to 567f361 Compare December 22, 2025 22:12
@jjnesbitt jjnesbitt force-pushed the backfill-jobs-command branch from 567f361 to de82b48 Compare December 22, 2025 22:43
@jjnesbitt jjnesbitt merged commit ca25758 into main Dec 22, 2025
5 checks passed
@jjnesbitt jjnesbitt deleted the backfill-jobs-command branch December 22, 2025 22:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant