Skip to content

fix(source-twilio): Resume messages from saved cursor#78358

Draft
devin-ai-integration[bot] wants to merge 3 commits into
masterfrom
devin/1779446220-fix-twilio-messages-state
Draft

fix(source-twilio): Resume messages from saved cursor#78358
devin-ai-integration[bot] wants to merge 3 commits into
masterfrom
devin/1779446220-fix-twilio-messages-state

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented May 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

Resolves https://github.com/airbytehq/oncall/issues/12689:

Fix the Twilio messages stream so incremental syncs resume from the saved global date_sent cursor instead of an older per-partition cursor.

How

  • Enable global_substream_cursor for messages, which uses one account-level time cursor for the stream and avoids continuing to track stale per-partition cursors.
  • Migrate existing mixed messages state that already contains a stream-level state.date_sent value into the global cursor shape expected by the concurrent cursor.
  • Update focused stream tests to cover the saved global state shape and a mixed legacy state where the global cursor is newer than partition state.
  • Bump source-twilio to 1.0.1 and add a changelog entry.

Declarative-First Evaluation

source-twilio is a low-code connector with custom components (language:manifest-only, cdk:low-code, plus components.py). The runtime behavior change uses the declarative DatetimeBasedCursor option global_substream_cursor, which is sufficient for future messages state emission. Existing mixed legacy state still needs normalization before the declarative cursor reads it, so this PR extends the existing CustomStateMigration; declarative alternatives such as RecordFilter, AddFields / RemoveFields, transformations, $ref overrides, and paginator/requester settings do not migrate persisted stream state.

Breaking Change Evaluation

Not breaking. The diff does not remove or rename streams, fields, primary keys, cursor fields, or config fields. The stream state shape changes to a global cursor, but the PR includes a state migration for the legacy mixed state shape, so users should not need manual migration action. Patch version 1.0.1 is appropriate; progressive rollout is disabled for this connector.

Test Coverage

Review guide

  1. airbyte-integrations/connectors/source-twilio/components.py
  2. airbyte-integrations/connectors/source-twilio/manifest.yaml
  3. airbyte-integrations/connectors/source-twilio/unit_tests/test_streams.py
  4. airbyte-integrations/connectors/source-twilio/metadata.yaml
  5. docs/integrations/sources/twilio.md

User Impact

Twilio messages incremental syncs should resume from the stream-level saved date_sent cursor and avoid re-reading old message history when stale partition state is present.

Can this PR be safely reverted and rolled back?

  • YES
  • NO

Original Devin session
Verification Devin session

Requested by: Rajath sinha (@rajathsinhaa)

@devin-ai-integration

devin-ai-integration Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@octavia-bot octavia-bot Bot marked this pull request as draft May 22, 2026 10:46
@octavia-bot

octavia-bot Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor

Note

📝 PR Converted to Draft

More info...

Thank you for creating this PR. As a policy to protect our engineers' time, Airbyte requires all PRs to be created first in draft status. Your PR has been automatically converted to draft status in respect for this policy.

As soon as your PR is ready for formal review, you can proceed to convert the PR to "ready for review" status by clicking the "Ready for review" button at the bottom of the PR page.

To skip draft status in future PRs, please include [ready] in your PR title or add the skip-draft-status label when creating your PR.

@github-actions

github-actions Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

  • 🛠️ Quick Fixes
    • /format-fix - Fixes most formatting issues.
    • /bump-version - Bumps connector versions, scraping changelog description from the PR title.
      • Bump types: patch (default), minor, major, major_rc, rc, promote.
      • The rc type is a smart default: applies minor_rc if stable, or bumps the RC number if already RC.
      • The promote type strips the RC suffix to finalize a release.
      • Example: /bump-version type=rc or /bump-version type=minor
    • /bump-progressive-rollout-version - Alias for /bump-version type=rc. Bumps with an RC suffix and enables progressive rollout.
  • ❇️ AI Testing and Review (internal link: AI-SDLC Docs):
    • /ai-prove-fix - Runs prerelease readiness checks, including testing against customer connections.
    • /ai-canary-prerelease - Rolls out prerelease to 5-10 connections for canary testing.
    • /ai-review - AI-powered PR review for connector safety and quality gates.
  • 📝 AI Documentation:
    • /ai-docs-review - AI-powered documentation review for PRs with connector changes.
    • /ai-create-docs-pr - Creates a documentation PR for connector changes, stacked on the current PR.
  • 🚀 Connector Releases:
    • /publish-connectors-prerelease - Publishes pre-release connector builds (tagged as {version}-preview.{git-sha}) for all modified connectors in the PR.
  • ☕️ JVM connectors:
    • /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
      Example: /update-connector-cdk-version connector=destination-bigquery
  • 🐍 Python connectors:
    • /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
    • /poe source example lock - Alias for /poe connector source-example lock.
    • /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
    • /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.
  • ⚙️ Admin commands:
    • /force-merge reason="<REASON>" - Force merges the PR using admin privileges, bypassing CI checks. Requires a reason.
      Example: /force-merge reason="CI is flaky, tests pass locally"
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

@github-actions

github-actions Bot commented May 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Deploy preview for airbyte-docs ready!

Project:airbyte-docs
Status: ✅  Deploy successful!
Preview URL:https://airbyte-docs-3ly0c6n8a-airbyte-growth.vercel.app
Latest Commit:f32b099

Deployed with vercel-action

@github-actions

github-actions Bot commented May 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

source-twilio Connector Test Results

25 tests   22 ✅  32s ⏱️
 2 suites   3 💤
 2 files     0 ❌

Results for commit f32b099.

♻️ This comment has been updated with latest results.

@devin-ai-integration @rajathsinhaa
fix(source-twilio): Migrate messages state to global cursor
f32b099 
Co-Authored-By: Rajath Sinha <rajath.sinha@airbyte.io>
@octavia-bot

octavia-bot Bot commented May 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🔍 AI Prove Fix session starting... Running readiness checks and testing against customer connections. View playbook

✅ Devin AI session created successfully!

@devin-ai-integration

devin-ai-integration Bot commented May 22, 2026 •
edited by airbyte-support-bot
Loading

Copy link
Copy Markdown
Contributor Author

Fix Validation Evidence

Status: Regression attempt 3 is running a targeted messages READ after attempt 2 hung during all-stream READ. Pre-release build succeeded for 1.0.1-preview.f32b099.

Current Findings

  • PR: fix(source-twilio): Resume messages from saved cursor #78358
  • Connector: source-twilio
  • Pre-release publish workflow: https://github.com/airbytehq/airbyte/actions/runs/26287051473
  • Pre-release tag expected: 1.0.1-preview.f32b099
  • Pre-flight: viability/safety checks passed from static review; no malicious patterns found.
  • Breaking/reversibility: no schema fields removed/renamed, no spec changes, no stream removals, and the only state-shape change includes a migration from mixed legacy messages state to global cursor state. Rollback appears safe because old versions can continue from a top-level date_sent cursor. Patch bump/changelog present.

Evidence Plan

Proving criteria: A regression comparison where target source-twilio starts the messages stream from the saved stream-level date_sent cursor, while control either re-reads older per-partition history or differs in a way that matches the intended cursor-state migration. At minimum, focused tests and regression checks must show no unexpected SPEC/CHECK/DISCOVER/READ regressions.

Disproving criteria: The target still resumes messages from stale per-partition state, emits an incompatible state shape, or introduces unexpected SPEC/CHECK/DISCOVER/READ differences unrelated to the cursor fix.

Testing strategy: Regression tests first and likely sufficient. source-twilio is a source connector, does not use forced OAuth write-back, and this fix is manifest/state-migration logic that comparison-mode regression tests can validate without modifying customer connections. Live connection pinning will only be considered if regression evidence is inconclusive and approval is available.

Ranked cases:

  1. GSM regression test credentials, all streams — default isolated comparison baseline.
  2. Existing Twilio connector live-test connection A, all streams — backup if GSM does not exercise messages cursor behavior.
  3. Existing Twilio connector live-test connection B with lookback config — backup with different config if case A is inconclusive.

Devin session

Regression attempt 1: https://github.com/airbytehq/airbyte-ops-mcp/actions/runs/26287096007

Attempt 1 result: failed before comparison because the control image airbyte/source-twilio:1.0.0 was unavailable. This did not exercise connector behavior.
Local targeted check: test_messages_prefers_global_cursor_over_partition_state passed (1 passed, 13 deselected) against the PR code.

Regression attempt 2: https://github.com/airbytehq/airbyte-ops-mcp/actions/runs/26287198348

Regression attempt 2 interim: SPEC, CHECK, and DISCOVER both succeeded with no regression against control 0.17.10; READ is still running.

Regression attempt 2 result: SPEC/CHECK/DISCOVER both succeeded with no regression, but all-stream READ hung and was cancelled before producing READ artifacts.
Regression attempt 3: https://github.com/airbytehq/airbyte-ops-mcp/actions/runs/26287707628 (targeted messages stream, debug logs enabled, control 0.17.10)

@devin-ai-integration

devin-ai-integration Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor Author

↪️ Triggering /ai-prove-fix per Hands-Free AI Triage Project triage next step.

Reason: Draft connector fix PR has connector tests passing and no prior prove-fix result, so validation is the next Hydra stage.

Devin session

@github-actions

github-actions Bot commented May 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Pre-release Connector Publish Started

Publishing pre-release build for connector source-twilio.
PR: #78358

Pre-release versions will be tagged as {version}-preview.f32b099
and are available for version pinning via the scoped_configuration API.

View workflow run
Pre-release Publish: SUCCESS ✅

Docker image (pre-release):
airbyte/source-twilio:1.0.1-preview.f32b099

Docker Hub: https://hub.docker.com/layers/airbyte/source-twilio/1.0.1-preview.f32b099

Registry JSON:

@devin-ai-integration

devin-ai-integration Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #78359, which contains the same final diff plus the required /ai-fix PR body sections and issue linkage.

Devin session

@devin-ai-integration

devin-ai-integration Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor Author

Fix Proven Successfully

Evidence Summary

  • Pre-release publish succeeded: https://github.com/airbytehq/airbyte/actions/runs/26287051473 (airbyte/source-twilio:1.0.1-preview.f32b099).
  • Static checks passed: the change is scoped to source-twilio messages cursor/state migration, with no schema field removals, spec changes, stream removals, or malicious patterns found.
  • Targeted local test passed: test_messages_prefers_global_cursor_over_partition_state (1 passed, 13 deselected).
  • Regression attempt 3 passed SPEC/CHECK/DISCOVER/READ for the messages stream against control airbyte/source-twilio:0.17.10: https://github.com/airbytehq/airbyte-ops-mcp/actions/runs/26287707628.
  • READ comparison emitted 20 messages records on both control and target. The target emitted a single global-cursor state (use_global_cursor: true) with state.date_sent, while control retained per-partition state. This matches the intended migration behavior.

Outcome
The fix is proven for prerelease readiness. Live connection pinning was not needed because the targeted source connector regression exercised messages READ behavior and the state-shape migration directly.

Next Steps
Proceed with normal review/merge. After merge, monitor the initial rollout for source-twilio incremental messages sync behavior and any state/READ regressions.

Detailed validation notes

Devin session

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rajathsinhaa Rajath sinha (rajathsinhaa) Awaiting requested review from rajathsinhaa

At least 1 approving review is required to merge this pull request.

Assignees

@rajathsinhaa Rajath sinha (rajathsinhaa)

Labels

connectors/source/twilio

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@octavia-squidington-iii @rajathsinhaa