Skip to content

feat: Add max_lookback_days parameter to Source.read() #632

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

devin-ai-integration[bot]
Copy link
Contributor

Added an optional max_lookback_days parameter to PyAirbyte's Source.read() implementation. When provided, a UTC start_date is calculated by subtracting the specified number of days from the current UTC datetime and injected into the connector's configuration at runtime. In this mode, state changes are not committed since the partial sync cannot be considered valid for being caught up to the current time.

Link to Devin run: https://app.devin.ai/sessions/8f2a97d15d474e0db83aac5bc20b3e18
Requested by: Aaron ("AJ") Steers ([email protected])

Copy link
Contributor Author

🤖 Devin AI Engineer

Original prompt from Aaron:

@Devin - I'd like to add an optional arg to PyAirbyte's Source.read() implementation. The arg would be "max_lookback_days" and would accept an integer or None, defaulting to None. When not provided (None) then there would be no change to behavior. When provided, then a utc start_date (datetime string) would be calculated backwards from utcnow and that would be injected into the connector's config at runtime. Also, in this mode would would not commit any state changes - since we can't consider the partial sync to be valid in terms of being caught up to the current time.

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add "(aside)" to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@aaronsteers
Copy link
Contributor

Closing for now in favor of:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant