Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(connector-builder): add full_resolve command to connector builder #442

Merged
merged 12 commits into from
Mar 27, 2025
+697 −2
Merged

feat(connector-builder): add full_resolve command to connector builder #442

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
56b88af
Add dynamic stream name to stream model
lazebnyi Mar 21, 2025
df8f72a
Add full-resolve-manifest command to connector builder
lazebnyi Mar 22, 2025
4e9e7da
Auto-fix lint and format issues
Mar 22, 2025
c757bbd
Fix mypy
lazebnyi Mar 22, 2025
f09a4ec
Merge branch 'lazebnyi/add-full-resolve-command-to-builder-handler' o…
lazebnyi Mar 22, 2025
da34498
Add generated streams to manifest
lazebnyi Mar 25, 2025
1d50394
Fix mypy
lazebnyi Mar 25, 2025
52de678
Update unit test manifest
lazebnyi Mar 25, 2025
97f4614
Update unit test manifest base url
lazebnyi Mar 25, 2025
f13b128
Add dynamic_streams_name only for full_resolve command
lazebnyi Mar 26, 2025
9ea30e0
Auto-fix lint and format issues
Mar 26, 2025
524e481
Add except clause for AirbyteTracedException
lazebnyi Mar 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update unit test manifest base url
@lazebnyi
lazebnyi committed Mar 25, 2025
commit 97f461426b8d2a091013f8e68bc2db3a46e3171a
374 changes: 374 additions & 0 deletions unit_tests/connector_builder/test.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,374 @@
Full diff:
{
'check': {'stream_names': ['lists'], 'type': 'CheckStream'},
'definitions': {'retriever': {'paginator': {'page_size': 2,
'page_size_option': {'field_name': 'page_size',
'inject_into': 'request_parameter'},
'page_token_option': {'inject_into': 'path',
'type': 'RequestPath'},
'pagination_strategy': {'cursor_value': '{{ '
'response._metadata.next '
'}}',
'page_size': 2,
'type': 'CursorPagination'},
'type': 'DefaultPaginator'},
'partition_router': {'cursor_field': 'item_id',
'type': 'ListPartitionRouter',
'values': ['0',
'1',
'2',
'3',
'4',
'5',
'6',
'7']},
'record_selector': {'extractor': {'field_path': ['result']}},
'requester': {'authenticator': {'api_token': '{{ '
'config.apikey '
'}}',
'type': 'BearerAuthenticator'},
'path': '/v3/marketing/lists',
'request_parameters': {'a_param': '10'}}}},
'dynamic_streams': [{'components_resolver': {'components_mapping': [{'field_path': ['name'],
'type': 'ComponentMappingDefinition',
'value': "parent_{{stream_slice['parent_id']}}_{{components_values['name']}}"},
{'field_path': ['retriever',
'requester',
'path'],
'type': 'ComponentMappingDefinition',
'value': '{{ '
"stream_slice['parent_id'] "
'}}/{{ '
"components_values['id'] "
'}}'}],
'retriever': {'paginator': {'type': 'NoPagination'},
'partition_router': {'parent_stream_configs': [{'parent_key': 'id',
'partition_field': 'parent_id',
'stream': {'name': 'parent',
'retriever': {'record_selector': {'extractor': {'field_path': [],
'type': 'DpathExtractor'},
'type': 'RecordSelector'},
'requester': {'authenticator': {'api_token': '{{ '
"config['api_key'] "
'}}',
'header': 'apikey',
'type': 'ApiKeyAuthenticator'},
'http_method': 'GET',
'path': '/parents',
'type': 'HttpRequester',
'url_base': 'https://api.test.com'},
'type': 'SimpleRetriever'},
'schema_loader': {'schema': {'$schema': 'http://json-schema.org/schema#',
'properties': {'id': {'type': 'integer'}},
'type': 'object'},
'type': 'InlineSchemaLoader'},
'type': 'DeclarativeStream'},
'type': 'ParentStreamConfig'}],
'type': 'SubstreamPartitionRouter'},
'record_selector': {'extractor': {'field_path': [],
'type': 'DpathExtractor'},
'type': 'RecordSelector'},
'requester': {'authenticator': {'api_token': '{{ '
"config['api_key'] "
'}}',
'header': 'apikey',
'type': 'ApiKeyAuthenticator'},
'http_method': 'GET',
'path': 'parent/{{ '
'stream_partition.parent_id '
'}}/items',
'type': 'HttpRequester',
'url_base': 'https://api.test.com',
'use_cache': True},
'type': 'SimpleRetriever'},
'type': 'HttpComponentsResolver'},
'name': 'TestDynamicStream',
'stream_template': {'name': '',
'primary_key': [],
'retriever': {'paginator': {'type': 'NoPagination'},
'record_selector': {'extractor': {'field_path': [],
'type': 'DpathExtractor'},
'type': 'RecordSelector'},
'requester': {'authenticator': {'api_token': '{{ '
"config['api_key'] "
'}}',
'header': 'apikey',
'type': 'ApiKeyAuthenticator'},
'http_method': 'GET',
'path': '',
'type': 'HttpRequester',
'url_base': 'https://api.test.com'},
'type': 'SimpleRetriever'},
'schema_loader': {'schema': {'$schema': 'http://json-schema.org/schema#',
'properties': {'ABC': {'type': 'number'},
'AED': {'type': 'number'}},
'type': 'object'},
'type': 'InlineSchemaLoader'},
'type': 'DeclarativeStream'},
'type': 'DynamicDeclarativeStream'}],
'spec': {'connection_specification': {'$schema': 'http://json-schema.org/draft-07/schema#',
'additionalProperties': True,
'properties': {},
'required': [],
'type': 'object'},
'type': 'Spec'},
'streams': [{'$parameters': {'name': 'stream_with_custom_requester',
'primary_key': 'id',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'dynamic_stream_name': None,
'name': 'stream_with_custom_requester',
'primary_key': 'id',
'retriever': {'$parameters': {'name': 'stream_with_custom_requester',
'primary_key': 'id',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'name': 'stream_with_custom_requester',
'paginator': {'$parameters': {'name': 'stream_with_custom_requester',
'primary_key': 'id',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'name': 'stream_with_custom_requester',
'page_size': 2,
'page_size_option': {'$parameters': {'name': 'stream_with_custom_requester',
'primary_key': 'id',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'field_name': 'page_size',
'inject_into': 'request_parameter',
'name': 'stream_with_custom_requester',
'primary_key': 'id',
'type': 'RequestOption',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'page_token_option': {'$parameters': {'name': 'stream_with_custom_requester',
'primary_key': 'id',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'inject_into': 'path',
'name': 'stream_with_custom_requester',
'primary_key': 'id',
'type': 'RequestPath',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'pagination_strategy': {'$parameters': {'name': 'stream_with_custom_requester',
'primary_key': 'id',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'cursor_value': '{{ '
'response._metadata.next '
'}}',
'name': 'stream_with_custom_requester',
'page_size': 2,
'primary_key': 'id',
'type': 'CursorPagination',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'primary_key': 'id',
'type': 'DefaultPaginator',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'partition_router': {'$parameters': {'name': 'stream_with_custom_requester',
'primary_key': 'id',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'cursor_field': 'item_id',
'name': 'stream_with_custom_requester',
'primary_key': 'id',
'type': 'ListPartitionRouter',
- 'url_base': 'https://api.sendgrid.com',
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/',
? +++++++++++ ^^^^
'values': ['0',
'1',
'2',
'3',
'4',
'5',
'6',
'7']},
'primary_key': 'id',
'record_selector': {'$parameters': {'name': 'stream_with_custom_requester',
'primary_key': 'id',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'extractor': {'$parameters': {'name': 'stream_with_custom_requester',
'primary_key': 'id',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'field_path': ['result'],
'name': 'stream_with_custom_requester',
'primary_key': 'id',
'type': 'DpathExtractor',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'name': 'stream_with_custom_requester',
'primary_key': 'id',
'type': 'RecordSelector',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'requester': {'$parameters': {'name': 'stream_with_custom_requester',
'primary_key': 'id',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'authenticator': {'$parameters': {'name': 'stream_with_custom_requester',
'primary_key': 'id',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'api_token': '{{ '
'config.apikey '
'}}',
'name': 'stream_with_custom_requester',
'primary_key': 'id',
'type': 'BearerAuthenticator',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'name': 'stream_with_custom_requester',
'path': '/v3/marketing/lists',
'primary_key': 'id',
'request_parameters': {'a_param': '10'},
'type': 'HttpRequester',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'type': 'SimpleRetriever',
- 'url_base': 'https://api.sendgrid.com'},
? ^^^^^^^^^^^^^
+ 'url_base': 'https://10.0.27.27/api/v1/'},
? +++++++++++ ^^^^
'type': 'DeclarativeStream',
- 'url_base': 'https://api.sendgrid.com'},
+ 'url_base': 'https://10.0.27.27/api/v1/'},
{'dynamic_stream_name': 'TestDynamicStream',
'name': 'parent_1_item_1',
'primary_key': [],
'retriever': {'paginator': {'type': 'NoPagination'},
'record_selector': {'extractor': {'field_path': [],
'type': 'DpathExtractor'},
'type': 'RecordSelector'},
'requester': {'authenticator': {'api_token': '{{ '
"config['api_key'] "
'}}',
'header': 'apikey',
'type': 'ApiKeyAuthenticator'},
'http_method': 'GET',
'path': '1/1',
'type': 'HttpRequester',
'url_base': 'https://api.test.com'},
'type': 'SimpleRetriever'},
'schema_loader': {'schema': {'$schema': 'http://json-schema.org/schema#',
'properties': {'ABC': {'type': 'number'},
'AED': {'type': 'number'}},
'type': 'object'},
'type': 'InlineSchemaLoader'},
'type': 'DeclarativeStream'},
{'dynamic_stream_name': 'TestDynamicStream',
'name': 'parent_1_item_2',
'primary_key': [],
'retriever': {'paginator': {'type': 'NoPagination'},
'record_selector': {'extractor': {'field_path': [],
'type': 'DpathExtractor'},
'type': 'RecordSelector'},
'requester': {'authenticator': {'api_token': '{{ '
"config['api_key'] "
'}}',
'header': 'apikey',
'type': 'ApiKeyAuthenticator'},
'http_method': 'GET',
'path': '1/2',
'type': 'HttpRequester',
'url_base': 'https://api.test.com'},
'type': 'SimpleRetriever'},
'schema_loader': {'schema': {'$schema': 'http://json-schema.org/schema#',
'properties': {'ABC': {'type': 'number'},
'AED': {'type': 'number'}},
'type': 'object'},
'type': 'InlineSchemaLoader'},
'type': 'DeclarativeStream'},
{'dynamic_stream_name': 'TestDynamicStream',
'name': 'parent_2_item_1',
'primary_key': [],
'retriever': {'paginator': {'type': 'NoPagination'},
'record_selector': {'extractor': {'field_path': [],
'type': 'DpathExtractor'},
'type': 'RecordSelector'},
'requester': {'authenticator': {'api_token': '{{ '
"config['api_key'] "
'}}',
'header': 'apikey',
'type': 'ApiKeyAuthenticator'},
'http_method': 'GET',
'path': '2/1',
'type': 'HttpRequester',
'url_base': 'https://api.test.com'},
'type': 'SimpleRetriever'},
'schema_loader': {'schema': {'$schema': 'http://json-schema.org/schema#',
'properties': {'ABC': {'type': 'number'},
'AED': {'type': 'number'}},
'type': 'object'},
'type': 'InlineSchemaLoader'},
'type': 'DeclarativeStream'},
{'dynamic_stream_name': 'TestDynamicStream',
'name': 'parent_2_item_2',
'primary_key': [],
'retriever': {'paginator': {'type': 'NoPagination'},
'record_selector': {'extractor': {'field_path': [],
'type': 'DpathExtractor'},
'type': 'RecordSelector'},
'requester': {'authenticator': {'api_token': '{{ '
"config['api_key'] "
'}}',
'header': 'apikey',
'type': 'ApiKeyAuthenticator'},
'http_method': 'GET',
'path': '2/2',
'type': 'HttpRequester',
'url_base': 'https://api.test.com'},
'type': 'SimpleRetriever'},
'schema_loader': {'schema': {'$schema': 'http://json-schema.org/schema#',
'properties': {'ABC': {'type': 'number'},
'AED': {'type': 'number'}},
'type': 'object'},
'type': 'InlineSchemaLoader'},
'type': 'DeclarativeStream'}],
'type': 'DeclarativeSource',
'version': '0.30.3',
}
44 changes: 22 additions & 22 deletions unit_tests/connector_builder/test_connector_builder_handler.py
View file
Original file line number Diff line number Diff line change
@@ -1437,23 +1437,23 @@ def test_full_resolve_manifest(valid_resolve_manifest_config_file):
"type": "RequestOption",
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
"$parameters": {
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
},
},
"page_token_option": {
"inject_into": "path",
"type": "RequestPath",
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
"$parameters": {
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
},
},
"pagination_strategy": {
@@ -1462,20 +1462,20 @@ def test_full_resolve_manifest(valid_resolve_manifest_config_file):
"page_size": 2,
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
"$parameters": {
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
},
},
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
"$parameters": {
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
},
},
"partition_router": {
@@ -1484,11 +1484,11 @@ def test_full_resolve_manifest(valid_resolve_manifest_config_file):
"cursor_field": "item_id",
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
"$parameters": {
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
},
},
"requester": {
@@ -1498,22 +1498,22 @@ def test_full_resolve_manifest(valid_resolve_manifest_config_file):
"api_token": "{{ config.apikey }}",
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
"$parameters": {
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
},
},
"request_parameters": {"a_param": "10"},
"type": "HttpRequester",
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
"$parameters": {
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
},
},
"record_selector": {
@@ -1522,40 +1522,40 @@ def test_full_resolve_manifest(valid_resolve_manifest_config_file):
"type": "DpathExtractor",
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
"$parameters": {
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
},
},
"type": "RecordSelector",
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
"$parameters": {
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
},
},
"type": "SimpleRetriever",
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
"$parameters": {
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
},
},
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
"$parameters": {
"name": "stream_with_custom_requester",
"primary_key": "id",
"url_base": "https://api.sendgrid.com",
"url_base": "https://10.0.27.27/api/v1/",
},
"dynamic_stream_name": None,
},