Skip to content

feat(cdk): add schema filter to DynamicSchemaLoader #550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
May 20, 2025

Conversation

lazebnyi
Copy link
Contributor

@lazebnyi lazebnyi commented May 17, 2025

What

Resolved: https://github.com/airbytehq/airbyte-internal-issues/issues/13045

In some cases, when using the dynamic schema loader endpoint, the provider may return all possible fields that could be present. To keep the schema up to date and ensure that the properties section only includes the fields that were actually requested, we need to filter the returned values accordingly.

How

Added filtering behavior to the schema loader using the schema_filter.

Summary by CodeRabbit

  • New Features

    • Added support for filtering fields in dynamic schema loading, allowing selective inclusion of fields in the generated schema using configurable filters.
  • Documentation

    • Updated descriptions to clarify schema loader behavior and the merging of multiple schema loaders.
  • Tests

    • Enhanced tests to verify the new schema filtering functionality.

@github-actions github-actions bot added the enhancement New feature or request label May 17, 2025
@lazebnyi
Copy link
Contributor Author

lazebnyi commented May 17, 2025

/autofix

Auto-Fix Job Info

This job attempts to auto-fix any linting or formating issues. If any fixes are made,
those changes will be automatically committed and pushed back to the PR.

Note: This job can only be run by maintainers. On PRs from forks, this command requires
that the PR author has enabled the Allow edits from maintainers option.

PR auto-fix job started... Check job output.

✅ Changes applied successfully.

Copy link
Contributor

coderabbitai bot commented May 17, 2025

📝 Walkthrough

Walkthrough

A new optional schema_filter property was introduced to the DynamicSchemaLoader component schema and its Python implementation, enabling filtering of fields included in dynamically loaded JSON schemas. The factory method was updated to instantiate and pass this filter when present. Tests were modified to verify the filtering behavior, and relevant docstrings were clarified.

Changes

File(s) Change Summary
airbyte_cdk/sources/declarative/declarative_component_schema.yaml Added optional schema_filter property to DynamicSchemaLoader schema, accepting RecordFilter or CustomRecordFilter.
airbyte_cdk/sources/declarative/models/declarative_component_schema.py Updated DeclarativeStream docstring for schema_loader; added optional schema_filter field to DynamicSchemaLoader.
airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py Modified create_dynamic_schema_loader to instantiate and pass schema_filter to DynamicSchemaLoader if present.
airbyte_cdk/sources/declarative/schema/dynamic_schema_loader.py Added schema_filter attribute and filtering logic to DynamicSchemaLoader; refactored _transform signature.
unit_tests/sources/declarative/schema/test_dynamic_schema_loader.py Updated test manifest and test data to include and verify the new schema_filter behavior.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant DeclarativeStream
    participant ModelToComponentFactory
    participant DynamicSchemaLoader
    participant SchemaFilter

    User->>DeclarativeStream: Define stream with DynamicSchemaLoader (with optional schema_filter)
    DeclarativeStream->>ModelToComponentFactory: Create DynamicSchemaLoader
    ModelToComponentFactory->>DynamicSchemaLoader: Instantiate (schema_filter passed if present)
    DynamicSchemaLoader->>DynamicSchemaLoader: get_json_schema()
    DynamicSchemaLoader->>DynamicSchemaLoader: Extract schema properties
    alt schema_filter is set
        DynamicSchemaLoader->>SchemaFilter: filter_records(properties)
        SchemaFilter-->>DynamicSchemaLoader: Filtered properties
    end
    DynamicSchemaLoader->>DynamicSchemaLoader: Transform filtered properties
    DynamicSchemaLoader-->>DeclarativeStream: Return filtered/transformed schema
Loading

Would you like me to help draft an example configuration snippet for the schema_filter usage or a more detailed explanation of the filtering logic? Wdyt?

Note

⚡️ AI Code Reviews for VS Code, Cursor, Windsurf

CodeRabbit now has a plugin for VS Code, Cursor and Windsurf. This brings AI code reviews directly in the code editor. Each commit is reviewed immediately, finding bugs before the PR is raised. Seamless context handoff to your AI code agent ensures that you can easily incorporate review feedback.
Learn more here.


Note

⚡️ Faster reviews with caching

CodeRabbit now supports caching for code and dependencies, helping speed up reviews. This means quicker feedback, reduced wait times, and a smoother review experience overall. Cached data is encrypted and stored securely. This feature will be automatically enabled for all accounts on May 16th. To opt out, configure Review - Disable Cache at either the organization or repository level. If you prefer to disable all data retention across your organization, simply turn off the Data Retention setting under your Organization Settings.
Enjoy the performance boost—your workflow just got faster.


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge Base: Disabled due to data retention organization setting

📥 Commits

Reviewing files that changed from the base of the PR and between b00ffc1 and 41ff258.

📒 Files selected for processing (1)
  • unit_tests/sources/declarative/schema/test_dynamic_schema_loader.py (2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • unit_tests/sources/declarative/schema/test_dynamic_schema_loader.py
⏰ Context from checks skipped due to timeout of 90000ms (9)
  • GitHub Check: Check: 'source-amplitude' (skip=false)
  • GitHub Check: Check: 'source-pokeapi' (skip=false)
  • GitHub Check: Check: 'source-shopify' (skip=false)
  • GitHub Check: Check: 'source-hardcoded-records' (skip=false)
  • GitHub Check: Pytest (Fast)
  • GitHub Check: SDM Docker Image Build
  • GitHub Check: Pytest (All, Python 3.11, Ubuntu)
  • GitHub Check: Pytest (All, Python 3.10, Ubuntu)
  • GitHub Check: Analyze (python)
✨ Finishing Touches
  • 📝 Generate Docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

‼️ IMPORTANT
Auto-reply has been disabled for this repository in the CodeRabbit settings. The CodeRabbit bot will not respond to your replies unless it is explicitly tagged.

  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
airbyte_cdk/sources/declarative/resolvers/components_resolver.py (1)

25-25: Introduce create_or_update flag in ComponentMappingDefinition
Great to see a control flag added for mapping behavior! Should we simplify the annotation to create_or_update: bool = False instead of Optional[bool], since the default is always False? wdyt?

airbyte_cdk/sources/declarative/models/declarative_component_schema.py (1)

2417-2419: The schema_filter description needs improvement, wdyt?

The description is currently just a placeholder ("placeholder"). Consider providing a more informative description that explains the purpose of this field, similar to how other filter fields are documented in this file.

-    schema_filter: Optional[Union[RecordFilter, CustomRecordFilter]] = Field(
-        None, description="placeholder", title="Schema Filter"
-    )
+    schema_filter: Optional[Union[RecordFilter, CustomRecordFilter]] = Field(
+        None, 
+        description="Filter applied to schema properties during schema generation. Properties will be included only if they satisfy the filter condition.",
+        title="Schema Filter"
+    )
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge Base: Disabled due to data retention organization setting

📥 Commits

Reviewing files that changed from the base of the PR and between fa8d54d and 3961f6c.

📒 Files selected for processing (5)
  • airbyte_cdk/sources/declarative/declarative_component_schema.yaml (1 hunks)
  • airbyte_cdk/sources/declarative/models/declarative_component_schema.py (2 hunks)
  • airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py (1 hunks)
  • airbyte_cdk/sources/declarative/resolvers/components_resolver.py (2 hunks)
  • airbyte_cdk/sources/declarative/schema/dynamic_schema_loader.py (4 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (2)
airbyte_cdk/sources/declarative/models/declarative_component_schema.py (1)
airbyte_cdk/sources/declarative/extractors/record_filter.py (1)
  • RecordFilter (17-49)
airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py (2)
airbyte_cdk/sources/declarative/schema/dynamic_schema_loader.py (1)
  • DynamicSchemaLoader (120-300)
airbyte_cdk/sources/declarative/models/declarative_component_schema.py (1)
  • DynamicSchemaLoader (2410-2439)
🪛 GitHub Actions: Linters
airbyte_cdk/sources/declarative/schema/dynamic_schema_loader.py

[error] 183-183: mypy error: Need type annotation for "filtered_properties" (hint: "filtered_properties: dict[, ] = ...") [var-annotated]

airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py

[error] 2436-2436: mypy error: Argument 1 to "_create_component_from_model" of "ModelToComponentFactory" has incompatible type "RecordFilter | CustomRecordFilter | None"; expected "BaseModel" [arg-type]

⏰ Context from checks skipped due to timeout of 90000ms (4)
  • GitHub Check: Check: 'source-amplitude' (skip=false)
  • GitHub Check: Check: 'source-shopify' (skip=false)
  • GitHub Check: Pytest (All, Python 3.11, Ubuntu)
  • GitHub Check: Pytest (All, Python 3.10, Ubuntu)
🔇 Additional comments (7)
airbyte_cdk/sources/declarative/resolvers/components_resolver.py (1)

38-38: Mirror create_or_update in ResolvedComponentMappingDefinition
Nice consistency between the definition and its resolved counterpart. Could you verify that downstream logic (e.g., in the component factory or resolver implementations) properly consumes this new flag? wdyt?

airbyte_cdk/sources/declarative/declarative_component_schema.yaml (1)

2321-2326: Add optional schema_filter to DynamicSchemaLoader schema
This looks like a solid enhancement for selective field inclusion. Can you confirm that omitting schema_filter still results in the expected “no filter” behavior and that it’s correctly wired in the loader implementation? wdyt?

airbyte_cdk/sources/declarative/models/declarative_component_schema.py (1)

2167-2168: Good improvement to the schema loader description

The updated description now clearly explains what happens when multiple schema loaders are used in a declarative stream, particularly that properties will be merged with precedence given to loaders defined earlier in the list.

airbyte_cdk/sources/declarative/schema/dynamic_schema_loader.py (4)

13-13: Appropriate import addition

The import for RecordFilter is correctly added and needed for the new schema filtering functionality.


130-130: Good addition of the schema_filter parameter

The optional schema_filter parameter is correctly added to the DynamicSchemaLoader class.


156-156: Correct application of filtering before transformation

The get_json_schema method now correctly applies filtering before transformation, which is the appropriate order.


183-189: Consider empty stream state for filter_records, wdyt?

The filter_records method is called with an empty dictionary as the stream_state. Is this intentional? If a more appropriate state should be used, consider passing it.

Given that this is a schema filtering context rather than a normal data stream context, is passing an empty dictionary for stream_state appropriate for your use case? This might be fine if schema filtering doesn't need stream state information, but it's worth confirming.

🧰 Tools
🪛 GitHub Actions: Linters

[error] 183-183: mypy error: Need type annotation for "filtered_properties" (hint: "filtered_properties: dict[, ] = ...") [var-annotated]

@lazebnyi
Copy link
Contributor Author

lazebnyi commented May 17, 2025

/autofix

Auto-Fix Job Info

This job attempts to auto-fix any linting or formating issues. If any fixes are made,
those changes will be automatically committed and pushed back to the PR.

Note: This job can only be run by maintainers. On PRs from forks, this command requires
that the PR author has enabled the Allow edits from maintainers option.

PR auto-fix job started... Check job output.

✅ Changes applied successfully.

@lazebnyi lazebnyi requested review from bnchrch and brianjlai May 17, 2025 02:50
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🔭 Outside diff range comments (1)
unit_tests/sources/declarative/schema/test_dynamic_schema_loader.py (1)

1-1: 💡 Verification agent

🧩 Analysis chain

Small formatting issue flagged by pipeline

The pipeline shows a Ruff formatting issue that needs to be fixed.


🏁 Script executed:

#!/bin/bash
# Run ruff format check to see what needs to be fixed
ruff format --check unit_tests/sources/declarative/schema/test_dynamic_schema_loader.py --diff

Length of output: 677


Add missing trailing comma in schema_filter condition

Could you add a comma at the end of the "condition" line so that Ruff’s formatting check passes? wdyt?

  • File: unit_tests/sources/declarative/schema/test_dynamic_schema_loader.py
    Around line 70:
-                    "condition": "{{ 'filtered_field' not in record }}"
+                    "condition": "{{ 'filtered_field' not in record }}",
🧰 Tools
🪛 GitHub Actions: Linters

[error] 1-1: Ruff formatting check failed. File would be reformatted. Run 'ruff format' to fix code style issues.

🧹 Nitpick comments (1)
unit_tests/sources/declarative/schema/test_dynamic_schema_loader.py (1)

338-355: Consider explicit verification of filtering behavior

The test verifies the schema content implicitly (by checking the final expected schema doesn't contain 'filtered_field'). Would it be clearer to add an explicit assertion that 'filtered_field' is not in the properties?

Something like:

assert "filtered_field" not in actual_catalog.streams[0].json_schema["properties"]

This would make the test's purpose more obvious and serve as documentation of what's being tested, wdyt?

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge Base: Disabled due to data retention organization setting

📥 Commits

Reviewing files that changed from the base of the PR and between 3961f6c and b00ffc1.

📒 Files selected for processing (4)
  • airbyte_cdk/sources/declarative/models/declarative_component_schema.py (2 hunks)
  • airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py (1 hunks)
  • airbyte_cdk/sources/declarative/schema/dynamic_schema_loader.py (4 hunks)
  • unit_tests/sources/declarative/schema/test_dynamic_schema_loader.py (2 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
  • airbyte_cdk/sources/declarative/schema/dynamic_schema_loader.py
  • airbyte_cdk/sources/declarative/models/declarative_component_schema.py
  • airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
🧰 Additional context used
🪛 GitHub Actions: Linters
unit_tests/sources/declarative/schema/test_dynamic_schema_loader.py

[error] 1-1: Ruff formatting check failed. File would be reformatted. Run 'ruff format' to fix code style issues.

⏰ Context from checks skipped due to timeout of 90000ms (9)
  • GitHub Check: Check: 'source-pokeapi' (skip=false)
  • GitHub Check: Check: 'source-hardcoded-records' (skip=false)
  • GitHub Check: Check: 'source-amplitude' (skip=false)
  • GitHub Check: Check: 'source-shopify' (skip=false)
  • GitHub Check: Pytest (All, Python 3.10, Ubuntu)
  • GitHub Check: Pytest (All, Python 3.11, Ubuntu)
  • GitHub Check: Pytest (Fast)
  • GitHub Check: SDM Docker Image Build
  • GitHub Check: Analyze (python)
🔇 Additional comments (2)
unit_tests/sources/declarative/schema/test_dynamic_schema_loader.py (2)

71-74: Schema filter implementation looks good!

The implementation adds a filter to exclude records with 'filtered_field' key, which aligns with the PR objective to filter properties in the dynamic schema loader. It's clear and concise.

Would adding a brief comment explaining the purpose of this schema filter make it more maintainable for future developers, wdyt?


397-397: Test data correctly updated to validate filtering

This new field entry is well placed to test if the schema_filter successfully excludes fields that shouldn't be included in the final schema. The test setup effectively validates that 'filtered_field' doesn't appear in the expected schema properties.

@lazebnyi
Copy link
Contributor Author

lazebnyi commented May 19, 2025

/autofix

Auto-Fix Job Info

This job attempts to auto-fix any linting or formating issues. If any fixes are made,
those changes will be automatically committed and pushed back to the PR.

Note: This job can only be run by maintainers. On PRs from forks, this command requires
that the PR author has enabled the Allow edits from maintainers option.

PR auto-fix job started... Check job output.

✅ Changes applied successfully.

Copy link
Contributor

@brianjlai brianjlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚢

@lazebnyi lazebnyi merged commit 4b73b46 into main May 20, 2025
27 of 29 checks passed
@lazebnyi lazebnyi deleted the lazebnyi/add-schema-filter branch May 20, 2025 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants