fix(source-twilio): Resume messages from saved cursor#78359
Draft
devin-ai-integration[bot] wants to merge 3 commits into
Draft
fix(source-twilio): Resume messages from saved cursor#78359devin-ai-integration[bot] wants to merge 3 commits into
devin-ai-integration[bot] wants to merge 3 commits into
Conversation
Co-Authored-By: Rajath Sinha <rajath.sinha@airbyte.io>
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Contributor
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
Contributor
|
Contributor
|
Deploy preview for airbyte-docs ready!
Deployed with vercel-action |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fix the Twilio
messagesstream so incremental syncs resume from the saved globaldate_sentcursor instead of an older per-partition cursor.Resolves https://github.com/airbytehq/oncall/issues/12688:
How
global_substream_cursorformessages, which uses one account-level time cursor for the stream and avoids continuing to track stale per-partition cursors.messagesstate that already contains a stream-levelstate.date_sentvalue into the global cursor shape expected by the concurrent cursor.source-twilioto1.0.1and add a changelog entry.Reproduction
messagesstate containing a newer globalstate.date_sentcursor and an older per-partition cursor.DateSent>equal to the global cursor value instead of the stale partition cursor.Declarative-First Evaluation
source-twiliois a low-code/manifest connector. The main behavior change is declarative:messagesnow uses the built-inDatetimeBasedCursorglobal_substream_cursoroption. I also evaluatedRecordFilter,AddFields/RemoveFields,DefaultPaginator,SubstreamPartitionRouter,HttpRequestererror handlers, transformations, and$refoverrides; those do not migrate already-persisted mixed state. The custom state migration change is limited to converting existing mixedmessagesstate withstate.date_sentinto the global cursor shape.Breaking Change Evaluation
Not breaking. This does not change schemas, primary keys, cursor field, stream availability, or required config. The connector state format change is handled by a state migration, so users do not need to reset or clear data.
Test Coverage
cd /home/ubuntu/repos/airbyte/airbyte-integrations/connectors/source-twilio/unit_tests && poetry run pytest test_streams.py::TestIncrementalTwilioStream::test_messages_prefers_global_cursor_over_partition_state -xcd /home/ubuntu/repos/airbyte/airbyte-integrations/connectors/source-twilio/unit_tests && poetry run pytest . -xcd /home/ubuntu/repos/airbyte/airbyte-integrations/connectors/source-twilio && poe test-unit-testscd /home/ubuntu/repos/airbyte/airbyte-integrations/connectors/source-twilio && poe format-checkpoe lint-checkwas not runnable because the shared manifest-only connector task definition currently has an unmatched quote inpoe-tasks/manifest-only-connector-tasks.toml. The manifest-only connector's configuredformat-checktask reports no format step.Review guide
airbyte-integrations/connectors/source-twilio/manifest.yamlairbyte-integrations/connectors/source-twilio/components.pyairbyte-integrations/connectors/source-twilio/unit_tests/test_streams.pyairbyte-integrations/connectors/source-twilio/metadata.yamldocs/integrations/sources/twilio.mdUser Impact
Twilio
messagesincremental syncs should resume from the stream-level saveddate_sentcursor and avoid re-reading old message history when stale partition state is present.Can this PR be safely reverted and rolled back?
Link to Devin session: https://app.devin.ai/sessions/de88b2a7c1e442cf946bc22cce02e065