🐛 Source Iterable: Fix schema issues in campaigns, email send, and subscriptions #67602
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. Helpful Resources
PR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
|
|
/bump-version changelog="Fix array schema definitions"
|
|
/format-fix
|
|
vai-airbyte
changed the title
|
Deploy preview for airbyte-docs ready! ✅ Preview Built with commit 90c38b7. |
matteogp
pushed a commit
that referenced
this pull request
Oct 10, 2025
…bscriptions (#67602) ## What Fixes array column serialization errors in source-iterable streams by properly defining array item types in JSONSchema definitions. **Problem:** Array columns (e.g., `campaigns.labels`, `emailListIds`, `channelIds`, `categories`) were being written as null values to the S3 Data Lake destination with `DESTINATION_SERIALIZATION_ERROR` in the `_airbyte_meta` column. **Root Cause:** Multiple stream schemas had array definitions with empty `items: {}`, which is ambiguous and prevents proper type mapping to destinations like Iceberg/Glue. **Affected Streams:** - `email_unsubscribe` (emailListIds, channelIds) - `email_send` (categories) - `email_send_skip` (categories) - `email_subscribe` (emailListIds) - `campaigns` (listIds, suppressionListIds, labels) ## How Updated JSONSchema definitions across affected stream schema files to specify explicit item types for all array fields: **Before:** ``` "emailListIds": { "type": ["null", "array"], "items": {} } ``` After: ``` "emailListIds": { "type": ["null", "array"], "items": { "type": "integer" } } ``` ## Review guide source_iterable/schemas/campaigns.json - Check listIds, suppressionListIds, labels source_iterable/schemas/email_unsubscribe.json - Check emailListIds, channelIds source_iterable/schemas/email_send.json - Check categories (in transactional data) source_iterable/schemas/email_send_skip.json - Check categories (in transactional data) source_iterable/schemas/email_subscribe.json - Check emailListIds Verify that all "items": {} instances have been replaced with proper type definitions. ## User Impact None expected - this is a schema clarification that aligns with actual data types. ## Can this PR be safely reverted and rolled back? - [X] YES 💚 - [ ] NO ❌ --------- Co-authored-by: Octavia Squidington III <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Fixes array column serialization errors in source-iterable streams by properly defining array item types in JSONSchema definitions.
Problem: Array columns (e.g.,
campaigns.labels,emailListIds,channelIds,categories) were being written as null values to the S3 Data Lake destination withDESTINATION_SERIALIZATION_ERRORin the_airbyte_metacolumn.Root Cause: Multiple stream schemas had array definitions with empty
items: {}, which is ambiguous and prevents proper type mapping to destinations like Iceberg/Glue.Affected Streams:
email_unsubscribe(emailListIds, channelIds)email_send(categories)email_send_skip(categories)email_subscribe(emailListIds)campaigns(listIds, suppressionListIds, labels)How
Updated JSONSchema definitions across affected stream schema files to specify explicit item types for all array fields:
Before:
After:
Review guide
source_iterable/schemas/campaigns.json - Check listIds, suppressionListIds, labels
source_iterable/schemas/email_unsubscribe.json - Check emailListIds, channelIds
source_iterable/schemas/email_send.json - Check categories (in transactional data)
source_iterable/schemas/email_send_skip.json - Check categories (in transactional data)
source_iterable/schemas/email_subscribe.json - Check emailListIds
Verify that all "items": {} instances have been replaced with proper type definitions.
User Impact
None expected - this is a schema clarification that aligns with actual data types.
Can this PR be safely reverted and rolled back?