Skip to content

support omitted streams #330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 18, 2025
Merged

support omitted streams #330

merged 2 commits into from
Mar 18, 2025

Conversation

Alex-Bair
Copy link
Member

@Alex-Bair Alex-Bair commented Mar 17, 2025

The activity_feed stream in source-greenhouse is an array API response that should be flattened, but the Airbyte connector emits it as-is. An abbreviated example response is below:

{
  "notes": [
    {
      "id": 1,
    }
  ],
  "emails": [
    {
      "id": 1,
    }
  ],
  "activities": [
    {
      "id": 1,
    },
    {
      "id": 2,
    }
  ]
}

The real solution is to fix the connector so individual objects from the response arrays are emitted instead of emitting the entire response as as single document. However, until the connector is fixed, omitting activity_feed from discovery is a temporary fix.

We previously had the capability to omit streams from discovery with selected_streams.json, but we changed it to specify streams that should be enabled & disabled everything else. To re-add the capability to completely omit streams from discovery, this PR adds omitted_streams.json.

I tested this locally with source-greenhouse, and activity_feed is omitted from discovery.

@Alex-Bair Alex-Bair marked this pull request as ready for review March 17, 2025 21:25
@Alex-Bair Alex-Bair merged commit c56664f into master Mar 18, 2025
46 of 57 checks passed
@Alex-Bair Alex-Bair deleted the bair/atf-support-omitted-streams branch March 18, 2025 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants