feat(cdk): add schema filter to DynamicSchemaLoader #550
Conversation
|
/autofix
|
|
Warning Rate limit exceeded@lazebnyi has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 4 minutes and 29 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe changes introduce a new optional Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant DeclarativeStream
participant DynamicSchemaLoader
participant RecordFilter
User->>DeclarativeStream: Request schema
DeclarativeStream->>DynamicSchemaLoader: get_json_schema()
DynamicSchemaLoader->>DynamicSchemaLoader: Extract schema properties
alt schema_filter is set
DynamicSchemaLoader->>RecordFilter: filter_records(properties)
RecordFilter-->>DynamicSchemaLoader: Filtered properties
else schema_filter not set
DynamicSchemaLoader->>DynamicSchemaLoader: Use all properties
end
DynamicSchemaLoader->>DynamicSchemaLoader: Apply schema transformations
DynamicSchemaLoader-->>DeclarativeStream: Return filtered/transformed schema
Would you like to see a diagram comparing the old and new flows for clarity, or is this high-level overview sufficient for your needs? Wdyt? ✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
airbyte_cdk/sources/declarative/resolvers/components_resolver.py (1)
25-25: Introducecreate_or_updateflag in ComponentMappingDefinition
Great to see a control flag added for mapping behavior! Should we simplify the annotation tocreate_or_update: bool = Falseinstead ofOptional[bool], since the default is alwaysFalse? wdyt?airbyte_cdk/sources/declarative/models/declarative_component_schema.py (1)
2417-2419: The schema_filter description needs improvement, wdyt?The description is currently just a placeholder ("placeholder"). Consider providing a more informative description that explains the purpose of this field, similar to how other filter fields are documented in this file.
- schema_filter: Optional[Union[RecordFilter, CustomRecordFilter]] = Field( - None, description="placeholder", title="Schema Filter" - ) + schema_filter: Optional[Union[RecordFilter, CustomRecordFilter]] = Field( + None, + description="Filter applied to schema properties during schema generation. Properties will be included only if they satisfy the filter condition.", + title="Schema Filter" + )
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge Base: Disabled due to data retention organization setting
📒 Files selected for processing (5)
airbyte_cdk/sources/declarative/declarative_component_schema.yaml(1 hunks)airbyte_cdk/sources/declarative/models/declarative_component_schema.py(2 hunks)airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py(1 hunks)airbyte_cdk/sources/declarative/resolvers/components_resolver.py(2 hunks)airbyte_cdk/sources/declarative/schema/dynamic_schema_loader.py(4 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (2)
airbyte_cdk/sources/declarative/models/declarative_component_schema.py (1)
airbyte_cdk/sources/declarative/extractors/record_filter.py (1)
RecordFilter(17-49)
airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py (2)
airbyte_cdk/sources/declarative/schema/dynamic_schema_loader.py (1)
DynamicSchemaLoader(120-300)airbyte_cdk/sources/declarative/models/declarative_component_schema.py (1)
DynamicSchemaLoader(2410-2439)
🪛 GitHub Actions: Linters
airbyte_cdk/sources/declarative/schema/dynamic_schema_loader.py
[error] 183-183: mypy error: Need type annotation for "filtered_properties" (hint: "filtered_properties: dict[, ] = ...") [var-annotated]
airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
[error] 2436-2436: mypy error: Argument 1 to "_create_component_from_model" of "ModelToComponentFactory" has incompatible type "RecordFilter | CustomRecordFilter | None"; expected "BaseModel" [arg-type]
⏰ Context from checks skipped due to timeout of 90000ms (4)
- GitHub Check: Check: 'source-amplitude' (skip=false)
- GitHub Check: Check: 'source-shopify' (skip=false)
- GitHub Check: Pytest (All, Python 3.11, Ubuntu)
- GitHub Check: Pytest (All, Python 3.10, Ubuntu)
🔇 Additional comments (7)
airbyte_cdk/sources/declarative/resolvers/components_resolver.py (1)
38-38: Mirrorcreate_or_updatein ResolvedComponentMappingDefinition
Nice consistency between the definition and its resolved counterpart. Could you verify that downstream logic (e.g., in the component factory or resolver implementations) properly consumes this new flag? wdyt?airbyte_cdk/sources/declarative/declarative_component_schema.yaml (1)
2321-2326: Add optionalschema_filterto DynamicSchemaLoader schema
This looks like a solid enhancement for selective field inclusion. Can you confirm that omittingschema_filterstill results in the expected “no filter” behavior and that it’s correctly wired in the loader implementation? wdyt?airbyte_cdk/sources/declarative/models/declarative_component_schema.py (1)
2167-2168: Good improvement to the schema loader descriptionThe updated description now clearly explains what happens when multiple schema loaders are used in a declarative stream, particularly that properties will be merged with precedence given to loaders defined earlier in the list.
airbyte_cdk/sources/declarative/schema/dynamic_schema_loader.py (4)
13-13: Appropriate import additionThe import for RecordFilter is correctly added and needed for the new schema filtering functionality.
130-130: Good addition of the schema_filter parameterThe optional schema_filter parameter is correctly added to the DynamicSchemaLoader class.
156-156: Correct application of filtering before transformationThe get_json_schema method now correctly applies filtering before transformation, which is the appropriate order.
183-189: Consider empty stream state for filter_records, wdyt?The filter_records method is called with an empty dictionary as the stream_state. Is this intentional? If a more appropriate state should be used, consider passing it.
Given that this is a schema filtering context rather than a normal data stream context, is passing an empty dictionary for stream_state appropriate for your use case? This might be fine if schema filtering doesn't need stream state information, but it's worth confirming.
🧰 Tools
🪛 GitHub Actions: Linters
[error] 183-183: mypy error: Need type annotation for "filtered_properties" (hint: "filtered_properties: dict[, ] = ...") [var-annotated]
airbyte_cdk/sources/declarative/schema/dynamic_schema_loader.py
Outdated
Show resolved
Hide resolved
airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
Outdated
Show resolved
Hide resolved
…byte-python-cdk into lazebnyi/add-schema-filter
|
/autofix
|
What
Resolved: https://github.com/airbytehq/airbyte-internal-issues/issues/13045
In some cases, when using the dynamic schema loader endpoint, the provider may return all possible fields that could be present. To keep the schema up to date and ensure that the
propertiessection only includes the fields that were actually requested, we need to filter the returned values accordingly.How
Added filtering behavior to the schema loader using the
schema_filter.Summary by CodeRabbit
New Features
Documentation