source-airtable-native: initial connector#3631
Merged
Conversation
Alex-Bair
added a commit
to estuary/flow
that referenced
this pull request
Dec 17, 2025
Documentation updates for estuary/connectors#3631. Marking `source-airtable` as deprecated since `source-airtable-native` is now the recommended way to capture data from Airtable.
Alex-Bair
added a commit
to estuary/flow
that referenced
this pull request
Dec 17, 2025
Documentation updates for estuary/connectors#3631. Marking `source-airtable` as deprecated since `source-airtable-native` is now the recommended way to capture data from Airtable.
JustinASmith
approved these changes
Dec 17, 2025
Contributor
JustinASmith
left a comment
JustinASmith
left a comment
There was a problem hiding this comment.
LGTM! Great write up throughout to explain the changes via commits, comments/docstrings, and the README.md.
Alex-Bair
force-pushed
the
bair/source-airtable-native
branch
from
3d4a812 to
ae5d78b
Compare
…tal states Previously, connector-initiated backfills only coordinated the backfill cutoff with the incremental cursor when state.inc was a single ResourceState.Incremental. When state.inc was a dict (multiple incremental tasks, e.g., REALTIME/LOOKBACK), the code would fall through to the else branch and wipe the entire state, resetting the incremental cursors to their initial values. This could cause the incremental cursors to jump ahead when a connector-initiated backfill began. If the backfill and incremental tasks always capture all fields, skipping the incremental cursor ahead would at worst cause a slight delay in when documents in the skipped portion were captured. However, if scheduled backfills do not capture all fields (like with formula field refreshes), fields for those skipped records could be permanently missed. This commit updates the CDK to handle the case where there are multiple incremental subtasks and single backfill task, and all incremental subtasks are using datetime cursors. The minimum incremental cursor across all incremental subtasks is now used as the cutoff for the single backfill task. The incremental cursors are left untouched, meaning they do not skip ahead and prevents the potential to skip data during a formula field refresh.
This commit introduces an initial implementation of a native Airtable capture connector. See the README.md for notable API features and connector design decisions.
We're still working on setting up an OAuth app in Airtable for the native connector. When that app's ready, we can uncomment the OAuth2Credentials option and allow users to authenticate via OAuth.
Alex-Bair
force-pushed
the
bair/source-airtable-native
branch
from
ae5d78b to
d98fa1b
Compare
Member
Author
|
I made a few changes post-approval:
|
Alex-Bair
added a commit
to estuary/flow
that referenced
this pull request
Dec 18, 2025
Documentation updates for estuary/connectors#3631. Marking `source-airtable` as deprecated since `source-airtable-native` is now the recommended way to capture data from Airtable.
Alex-Bair
added a commit
to estuary/flow
that referenced
this pull request
Dec 18, 2025
Documentation updates for estuary/connectors#3631. Marking `source-airtable` as deprecated since `source-airtable-native` is now the recommended way to capture data from Airtable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
This PR's scope includes:
IncrementalJSONProcessorto support passing in a validation context.datetimecursor among all incremental subtasks as thecutofffor the single backfill task.source-airtable-nativeconnector. See theREADME.mdfor notable API behaviors and connector design decisions.Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
Documentation needs created for the new connector.
Notes for reviewers:
Tested on a local stack. Confirmed:
I was able to create a test Airtable account, but the Free plan has a limit of 1,000 API requests per month. Our CI workflow runs frequently enough that we'd burn through those 1,000 API requests very quickly if we had discover or capture snapshot tests, then we'd be left with failing tests for the rest of the month. Instead, I opted to not have discover or capture snapshot tests.