Migration of a connection's Jobs
to Timeline
events
#46319
nataliekwong
started this conversation in
Guides & Tutorials
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Connection Timeline
As of v1.1.0, we have released the Connection Timeline feature.
This is a big change to our UI - allowing you to review not only your sync history, but also related connection activities to give you additional transparency. The "Timeline" replaces "Job History" for all connections.
Backfilling
If you are upgrading to v1.1.0 and beyond, you may want to backfill your existing job history into the new Timeline. This guide explains how to add those jobs to the Timeline.
Backfilling can be done by calling this migration endpoint:
/v1/connections/events/backfill
This is a permission controlled API which allows users to configure the request with the time window of jobs they want to backfill.
Example request:
Users can customize the time frame parameters (start time and end time) of jobs to be backfilled. Please note that as of now, we do not support deduplication in the time event table. If you are running the migration multiple times, or running the migration with a time window has any overlap with the existing events, duplicate events will be backfilled.
Only admin users have the permission to fulfill this request.
Cloud users
Cloud users have already been migrated to the Timeline and backfilled with data from 2024-06-20 and on. To see additional jobs created before this date, reach out to Airbyte Support for assistance in additional backfilling.
Supported Events
We support backfill jobs including sync, refresh, and clear (formerly reset). Using the final job status (succeeded, cancelled, failed, incomplete), we will log the following events on the Timeline:
Notes
We save manually started events (x_started events), but these events will not be backfilled because we cannot determine which jobs were manually triggered by just looking at Jobs table. Similarly, when a job was manually cancelled, we don’t know who performed the cancellation by just looking at the Jobs table. Therefore, when we backfill the x_cancelled events, the associated user ID will be missing (i.e. always null).
Beta Was this translation helpful? Give feedback.
All reactions