feat: Add HTTP Server integration with dynamic endpoints #2241

matarpeles · 2025-10-05T07:21:40Z

User description

…d support

This commit introduces a universal HTTP REST API integration for Ocean that can connect to any custom API without requiring custom development.

Key Features:

Universal HTTP connectivity with configurable authentication (Bearer, Basic, API Key, None)
Flexible pagination support (offset/limit, page/size, cursor-based, none)
Dynamic path parameters with API-discovered values
Endpoint-as-kind for granular tracking in Port's UI
Smart data extraction with JQ data_path expressions
Built-in caching and rate limiting via Ocean framework
Backward compatible with legacy selector format

Integration Components:

http_server/client.py: HTTP client with auth, pagination, and caching
http_server/overrides.py: Custom Pydantic models for configuration
main.py: Generic resync handler supporting dynamic kinds
integration.py: Integration class with custom config
initialize_client.py: Client factory with rate limiting

Examples:

Slack API integration (users, channels, workspace)
Dynamic endpoint resolution examples
Multiple authentication method examples

Documentation:

Comprehensive README with quick start and examples
VS Code launch configuration for debugging
Example configurations for common use cases

This integration is designed to work seamlessly with the Custom API Integration Wizard for AI-powered setup.

Description

What -

Why -

How -

Type of change

Please leave one option from the following and delete the rest:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
New Integration (non-breaking change which adds a new integration)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Non-breaking change (fix of existing functionality that will not change current behavior)
Documentation (added/updated documentation)

All tests should be run against the port production environment(using a testing org).

Core testing checklist

Integration able to create all default resources from scratch
Resync finishes successfully
Resync able to create entities
Resync able to update entities
Resync able to detect and delete entities
Scheduled resync able to abort existing resync and start a new one
Tested with at least 2 integrations from scratch
Tested with Kafka and Polling event listeners
Tested deletion of entities that don't pass the selector

Integration testing checklist

Preflight checklist

Handled rate limiting
Handled pagination
Implemented the code in async
Support Multi account

Screenshots

Include screenshots from your environment showing how the resources of the integration will look.

API Documentation

Provide links to the API documentation used for this integration.

PR Type

Enhancement

Description

Add universal HTTP REST API integration with dynamic endpoints
Support multiple authentication methods (Bearer, Basic, API Key, None)
Implement flexible pagination patterns (offset, page, cursor, none)
Enable dynamic path parameters with API-discovered values

Diagram Walkthrough

flowchart LR
  A["HTTP Client"] --> B["Authentication Layer"]
  B --> C["Pagination Handler"]
  C --> D["Dynamic Endpoint Resolver"]
  D --> E["Data Extraction"]
  E --> F["Port Entity Mapping"]
  G["Ocean Framework"] --> A
  H["JQ Processor"] --> E

File Walkthrough

Relevant files

Configuration changes

4 files

debug.py `Add debug entry point for VS Code`	+11/-0
launch.json `Add VS Code debug configurations`	+66/-0
spec.yaml `Define integration specification and configuration schema`	+71/-0
Makefile `Add development and build automation`	+70/-0

Miscellaneous

1 files

__init__.py `Initialize HTTP Server integration package`	+3/-0

Enhancement

5 files

client.py `Implement HTTP client with auth and pagination`	+351/-0
overrides.py `Define custom Pydantic models for configuration`	+49/-0
initialize_client.py `Create client factory with Ocean configuration`	+25/-0
integration.py `Define integration class with custom config`	+16/-0
main.py `Implement resync handlers with dynamic endpoints`	+278/-0

Tests

2 files

__init__.py `Initialize test package`	+3/-0
test_client.py `Add basic client initialization tests`	+51/-0

Documentation

12 files

entraid-blueprints.json `Define Entra ID blueprints for example`	+262/-0
entraid-port-app-config.yml `Configure Entra ID integration example`	+106/-0
port-app-config.yml `Define generic API resource configuration`	+121/-0
CHANGELOG.md `Add initial changelog entry`	+24/-0
README.md `Comprehensive documentation with examples and usage`	+614/-0
config-api-key.env `Example API key authentication configuration`	+31/-0
config-basic-auth.env `Example basic authentication configuration`	+32/-0
config-bearer-token.env `Example bearer token authentication configuration`	+30/-0
config-no-auth.env `Example no authentication configuration`	+26/-0
dynamic-endpoints.yaml `Example dynamic path parameter configuration`	+49/-0
slack-integration.env `Example Slack API integration configuration`	+24/-0
slack-integration.yaml `Example Slack API resource mapping`	+66/-0

Dependencies

1 files

pyproject.toml `Define project dependencies and build configuration`	+118/-0

…d support This commit introduces a universal HTTP REST API integration for Ocean that can connect to any custom API without requiring custom development. Key Features: - Universal HTTP connectivity with configurable authentication (Bearer, Basic, API Key, None) - Flexible pagination support (offset/limit, page/size, cursor-based, none) - Dynamic path parameters with API-discovered values - Endpoint-as-kind for granular tracking in Port's UI - Smart data extraction with JQ data_path expressions - Built-in caching and rate limiting via Ocean framework - Backward compatible with legacy selector format Integration Components: - http_server/client.py: HTTP client with auth, pagination, and caching - http_server/overrides.py: Custom Pydantic models for configuration - main.py: Generic resync handler supporting dynamic kinds - integration.py: Integration class with custom config - initialize_client.py: Client factory with rate limiting Examples: - Slack API integration (users, channels, workspace) - Dynamic endpoint resolution examples - Multiple authentication method examples Documentation: - Comprehensive README with quick start and examples - VS Code launch configuration for debugging - Example configurations for common use cases This integration is designed to work seamlessly with the Custom API Integration Wizard for AI-powered setup.

Copilot

Pull Request Overview

This PR introduces a comprehensive generic HTTP REST API integration for Ocean that enables users to connect to any custom API without requiring custom development. The integration provides universal HTTP connectivity with configurable authentication, pagination, and dynamic endpoint resolution.

Universal HTTP REST API integration with flexible authentication (Bearer, Basic, API Key, None)
Dynamic path parameter resolution using API-discovered values for nested endpoints
Smart data extraction with JQ expressions and endpoint-as-kind tracking for granular UI visibility

Reviewed Changes

Copilot reviewed 25 out of 26 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`integrations/http-server/main.py`	Core resync handler with dynamic endpoint resolution and JQ data extraction
`integrations/http-server/http_server/client.py`	HTTP client with authentication, pagination, and Ocean framework integration
`integrations/http-server/http_server/overrides.py`	Pydantic models for configuration with path parameters and selector extensions
`integrations/http-server/integration.py`	Integration class definition using custom configuration models
`integrations/http-server/initialize_client.py`	Client factory for HTTP server instances
`integrations/http-server/pyproject.toml`	Project configuration and dependencies
`integrations/http-server/tests/test_client.py`	Basic test coverage for client initialization
`integrations/http-server/README.md`	Comprehensive documentation with examples and configuration guide
`integrations/http-server/examples/`	Configuration examples for different authentication methods
`integrations/http-server/.port/`	Port configuration files including spec and blueprints
`.vscode/launch.json`	VS Code debugging configurations for the integration

Comments suppressed due to low confidence (1)

integrations/http-server/main.py:1

Function query_api_for_parameters is referenced on line 150 but not defined in this file. This will cause a NameError at runtime.

"""

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-05T07:22:22Z

integrations/http-server/http_server/overrides.py

+    field: str = Field(description="JQ expression to extract parameter value from each record")
+    filter: Optional[str] = Field(default=None, description="JQ boolean expression to filter records")


The ApiPathParameter model defines fields like field and filter but the code in main.py expects a search attribute and property attribute instead. This mismatch between the model definition and usage will cause AttributeError at runtime.

Suggested change

field: str = Field(description="JQ expression to extract parameter value from each record")

filter: Optional[str] = Field(default=None, description="JQ boolean expression to filter records")

property: str = Field(description="JQ expression to extract parameter value from each record")

search: Optional[str] = Field(default=None, description="JQ boolean expression to filter records")

Copilot · 2025-10-05T07:22:23Z

integrations/http-server/main.py

+    search_query = param_config.search
+    property_path = param_config.property


Accessing param_config.search and param_config.property but the ApiPathParameter model in overrides.py defines field and filter attributes instead. This will cause AttributeError.

Suggested change

search_query = param_config.search

property_path = param_config.property

search_query = param_config.filter

property_path = param_config.field

qodo-merge-pro · 2025-10-05T07:22:26Z

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
⚪	Credential leakage via shared client Description: The HTTP client sets global headers and auth on a shared singleton client (`http_async_client`), which may leak Authorization or API key credentials across different integration instances or requests if reused, causing cross-tenant credential exposure. client.py [52-71] Referred Code self.client.headers["User-Agent"] = "Port-Ocean-HTTP-Integration/1.0" # Configure authentication if self.auth_type == "bearer_token": token = self.auth_config.get("api_token") if token: self.client.headers["Authorization"] = f"Bearer {token}" elif self.auth_type == "api_key": api_key = self.auth_config.get("api_key") key_header = self.auth_config.get("api_key_header", "X-API-Key") if api_key and key_header: self.client.headers[key_header] = api_key elif self.auth_type == "basic": username = self.auth_config.get("username") password = self.auth_config.get("password") if username and password: self.client.auth = httpx.BasicAuth(username, password)
	Insecure dynamic property access Description: Property extraction for dynamic path parameters accepts arbitrary `property_path` without strict validation or whitelisting, which could lead to unintended data exposure if misconfigured to read sensitive fields. main.py [167-229] Referred Code """Query Port entities using Ocean's existing Port client with validation""" search_query = param_config.search property_path = param_config.property try: # Convert our search query to the format expected by Ocean's Port client query_dict = { "combinator": search_query.combinator, "rules": [ { "property": rule.property, "operator": rule.operator, "value": rule.value } for rule in search_query.rules ] } logger.info(f"Searching Port entities with query: {query_dict}") ... (clipped 42 lines)
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
⚪	No custom compliance provided Follow the guide to enable custom compliance check.
Update

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

qodo-merge-pro · 2025-10-05T07:23:47Z

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
General	✅ ~~Remove unused entity querying functions~~ Suggestion Impact: The commit added the missing query_api_for_parameters function that queries an API and extracts parameter values using jq expressions, aligning with the suggestion’s intent to implement this function. It did not remove query_port_entities, but the critical part (adding query_api_for_parameters) was implemented. code diff: +async def query_api_for_parameters(param_config: ApiPathParameter) -> List[str]: + """Query an API to get values for a path parameter""" + http_client = init_client() + + logger.info(f"Querying API for parameter values from {param_config.endpoint}") + + try: + async for batch in http_client.fetch_paginated_data( + endpoint=param_config.endpoint, + method=param_config.method, + query_params=param_config.query_params, + headers=param_config.headers, + ): + # Extract values using JQ expression + values = [] + for item in batch: + extracted_value = await extract_data_with_jq(item, param_config.field) + if extracted_value is not None: + # Apply optional filter + if param_config.filter: + filter_result = await extract_data_with_jq(item, param_config.filter) + if filter_result is True: + values.append(str(extracted_value)) + else: + values.append(str(extracted_value)) + + if values: + return values + + except Exception as e: + logger.error(f"Error querying API for parameter values from {param_config.endpoint}: {str(e)}") + + return [] Replace the unused `query_port_entities` function with the missing `query_api_for_parameters` implementation to fix a critical runtime error. integrations/http-server/main.py [166-192] -async def query_port_entities(param_config: ApiPathParameter) -> List[str]: - """Query Port entities using Ocean's existing Port client with validation""" - - search_query = param_config.search - property_path = param_config.property - +async def query_api_for_parameters(param_config: ApiPathParameter) -> List[str]: + """Query an API endpoint to discover values for a path parameter.""" + http_client = init_client() + endpoint = param_config.endpoint + method = param_config.method + query_params = param_config.query_params + headers = param_config.headers + jq_field = param_config.field + jq_filter = param_config.filter + + logger.info( + f"Discovering path parameter values from endpoint: {method} {endpoint}" + ) + + all_values = [] try: - # Convert our search query to the format expected by Ocean's Port client - query_dict = { - "combinator": search_query.combinator, - "rules": [ - { - "property": rule.property, - "operator": rule.operator, - "value": rule.value - } - for rule in search_query.rules - ] - } - - logger.info(f"Searching Port entities with query: {query_dict}") - - # Use Ocean's built-in Port client - entities = await ocean.port_client.search_entities( - user_agent_type=UserAgentType.exporter, - query=query_dict + async for batch in http_client.fetch_paginated_data( + endpoint=endpoint, + method=method, + query_params=query_params, + headers=headers, + ): + for item in batch: + # Apply filter if provided + if jq_filter: + filter_result = await extract_data_with_jq(item, jq_filter) + if not filter_result: + continue + + # Extract field value + value = await extract_data_with_jq(item, jq_field) + if value is not None: + all_values.append(value) + + logger.info( + f"Discovered {len(all_values)} values for path parameter. " + f"First 5: {all_values[:5]}{'...' if len(all_values) > 5 else ''}" ) + return all_values + except Exception as e: + logger.error(f"Failed to discover path parameter values: {e}") + return [] + `[Suggestion processed]` Suggestion importance[1-10]: 10 __ Why: The PR is broken as it calls an undefined function `query_api_for_parameters`, and this suggestion provides the missing implementation while correctly identifying that the existing `query_port_entities` function is unused and should be replaced.	High
General	✅ ~~Enable support for multiple path parameters~~ Suggestion Impact: The commit removed the in-file single-parameter resolver and imported resolve_dynamic_endpoints from a new module, implying an updated resolver. It also deleted the old single-parameter logic that warned on multiple parameters, aligning with the suggestion to support multiple parameters. code diff: +from http_server.overrides import HttpServerResourceConfig +from http_server.endpoint_resolver import resolve_dynamic_endpoints @ocean.on_resync() @@ -38,34 +22,25 @@ logger.info(f"Starting resync for kind (endpoint): {kind}") http_client = init_client() resource_config = cast(HttpServerResourceConfig, event.resource_config) - + selector = resource_config.selector - + # The kind IS the endpoint path (e.g., "/api/v1/users") # Check if endpoint has path parameters that need resolution endpoints = await resolve_dynamic_endpoints(selector, kind) - + logger.info(f"Resolved {len(endpoints)} endpoints to call for kind: {kind}") - - # Extract method, query_params, headers, data_path from selector (handle both formats) - query = getattr(selector, 'query', None) - if isinstance(query, dict): - # Old format: fields inside query dict - method = query.get('method', 'GET') - query_params = query.get('query_params', {}) - headers = query.get('headers', {}) - data_path = query.get('data_path') - else: - # New format: direct fields on selector - method = getattr(selector, 'method', 'GET') - query_params = getattr(selector, 'query_params', None) or {} - headers = getattr(selector, 'headers', None) or {} - data_path = getattr(selector, 'data_path', None) - + + # Extract method, query_params, headers, data_path from selector + method = getattr(selector, "method", "GET") + query_params = getattr(selector, "query_params", None) or {} + headers = getattr(selector, "headers", None) or {} + data_path = getattr(selector, "data_path", None) + # Call each resolved endpoint for endpoint in endpoints: logger.info(f"Fetching data from: {method} {endpoint}") - + try: async for batch in http_client.fetch_paginated_data( endpoint=endpoint, @@ -74,205 +49,34 @@ headers=headers, ): logger.info(f"Received {len(batch)} records from {endpoint}") - + # If data_path is specified, extract the array from each response if data_path: processed_batch = [] for item in batch: - extracted_data = await extract_data_with_jq(item, data_path) - if isinstance(extracted_data, list): - processed_batch.extend(extracted_data) - elif extracted_data is not None: - processed_batch.append(extracted_data) - + try: + # Use Ocean's built-in JQ processor + extracted_data = await ocean.app.integration.entity_processor._search( # type: ignore[attr-defined] + item, data_path + ) + if isinstance(extracted_data, list): + processed_batch.extend(extracted_data) + elif extracted_data is not None: + processed_batch.append(extracted_data) + except Exception as e: + logger.error( + f"Error extracting data with JQ path '{data_path}': {e}" + ) + continue + if processed_batch: - logger.info(f"Extracted {len(processed_batch)} items using data_path: {data_path}") + logger.info( + f"Extracted {len(processed_batch)} items using data_path: {data_path}" + ) yield processed_batch else: yield batch - + except Exception as e: logger.error(f"Error fetching data from {endpoint}: {str(e)}") continue - - # Note: No need to close Ocean's singleton HTTP client - - -async def resolve_dynamic_endpoints(selector: HttpServerSelector, kind: str) -> List[str]: - """Resolve dynamic endpoints using Ocean's Port client with validation - - Args: - selector: The resource selector configuration - kind: The endpoint path (e.g., "/api/v1/users" or "/api/v1/teams/{team_id}") - - Returns: - List of resolved endpoint URLs - """ - # The kind IS the endpoint - endpoint = kind - - # Get path_parameters from selector if they exist - query = getattr(selector, 'query', None) - if isinstance(query, dict): - # Old format: path_parameters inside query dict - path_parameters = query.get('path_parameters', {}) - else: - # New format: path_parameters as direct field on selector - path_parameters = getattr(selector, 'path_parameters', None) or {} - - if not endpoint: - logger.error("Kind (endpoint) is empty") - return [] - - # Find path parameters in endpoint template - param_names = re.findall(r'\{(\w+)\}', endpoint) - - if not param_names: - return [endpoint] - - # Validate that all parameters are configured - missing_params = [name for name in param_names if name not in path_parameters] - if missing_params: - logger.error(f"Missing configuration for path parameters: {missing_params}") - return [endpoint] - - # For now, handle single parameter (can extend for multiple later) - if len(param_names) > 1: - logger.warning( - f"Multiple path parameters detected: {param_names}. " - "Only the first parameter will be resolved." - ) - - param_name = param_names[0] - param_config = path_parameters[param_name] - - # Query API for parameter values - parameter_values = await query_api_for_parameters(param_config) - - if not parameter_values: - logger.error(f"No valid values found for path parameter '{param_name}'") - return [] - - # Generate resolved endpoints - resolved_endpoints = [] - for value in parameter_values: - resolved_endpoint = endpoint.replace(f"{{{param_name}}}", str(value)) - resolved_endpoints.append(resolved_endpoint) - - logger.info(f"Resolved {len(resolved_endpoints)} endpoints from parameter '{param_name}'") - return resolved_endpoints Extend the dynamic endpoint resolution logic to support multiple path parameters by iteratively resolving each parameter and generating all endpoint combinations. integrations/http-server/main.py [139-150] -# For now, handle single parameter (can extend for multiple later) -if len(param_names) > 1: - logger.warning( - f"Multiple path parameters detected: {param_names}. " - "Only the first parameter will be resolved." - ) +# Iteratively resolve all path parameters +resolved_endpoints = [endpoint] +for param_name in param_names: + param_config = path_parameters.get(param_name) + if not param_config: + logger.error(f"Missing configuration for path parameter: {param_name}") + return [] # Stop if any parameter is not configured -param_name = param_names[0] -param_config = path_parameters[param_name] + # Query API for parameter values + parameter_values = await query_api_for_parameters(param_config) + if not parameter_values: + logger.warning(f"No values found for path parameter '{param_name}', skipping endpoint resolution for it.") + return [] -# Query API for parameter values -parameter_values = await query_api_for_parameters(param_config) + # Generate new set of endpoints with the resolved parameter + newly_resolved_endpoints = [] + for current_endpoint in resolved_endpoints: + for value in parameter_values: + newly_resolved_endpoints.append(current_endpoint.replace(f"{{{param_name}}}", str(value))) + resolved_endpoints = newly_resolved_endpoints +logger.info(f"Resolved {len(resolved_endpoints)} final endpoints from parameters: {param_names}") +return resolved_endpoints + `[Suggestion processed]` Suggestion importance[1-10]: 7 __ Why: The suggestion proposes a significant feature enhancement to support multiple dynamic path parameters, which would greatly improve the integration's capability, though the current implementation is not buggy for single parameters.	Medium
High-level	The dynamic endpoint feature is incomplete The dynamic path parameter feature is broken because `main.py` calls a non-existent function `query_api_for_parameters`. The implementation must be completed by creating this function and removing the unused `query_port_entities` function. Examples: integrations/http-server/main.py [150] parameter_values = await query_api_for_parameters(param_config) integrations/http-server/main.py [166-252] async def query_port_entities(param_config: ApiPathParameter) -> List[str]: """Query Port entities using Ocean's existing Port client with validation""" search_query = param_config.search property_path = param_config.property try: # Convert our search query to the format expected by Ocean's Port client query_dict = { "combinator": search_query.combinator, ... (clipped 77 lines) Solution Walkthrough: Before: # in integrations/http-server/main.py async def resolve_dynamic_endpoints(selector: HttpServerSelector, kind: str) -> List[str]: # ... logic to find parameter name and config ... param_name = param_names[0] param_config = path_parameters[param_name] # This function is called but does NOT exist in the codebase parameter_values = await query_api_for_parameters(param_config) # ... logic to generate endpoints from values ... return resolved_endpoints # This function exists but is UNUSED and incompatible with the feature's design async def query_port_entities(param_config: ApiPathParameter) -> List[str]: # ... logic to query Port entities, not the target API # uses param_config.search which is not in the ApiPathParameter model ... After: # in integrations/http-server/main.py async def query_api_for_parameters(param_config: ApiPathParameter) -> List[str]: """Queries the discovery API to get values for a path parameter.""" http_client = init_client() values = [] async for batch in http_client.fetch_paginated_data(...): for item in batch: # Apply filter and extract field using JQ value = await extract_data_with_jq(item, param_config.field) if value is not None: values.append(str(value)) return values async def resolve_dynamic_endpoints(selector: HttpServerSelector, kind: str) -> List[str]: # ... logic to find parameter name and config ... param_name = param_names[0] param_config = path_parameters[param_name] # Now calls the correctly implemented function parameter_values = await query_api_for_parameters(param_config) # ... logic to generate endpoints from values ... return resolved_endpoints # The unused `query_port_entities` function should be removed. Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a critical bug where the dynamic path parameter feature is broken due to a call to the non-existent function `query_api_for_parameters` in `main.py`, rendering a key feature of the PR non-functional.	High
Possible issue	Fix incorrect response data wrapping Modify `_extract_items_from_response` to return the raw response data without wrapping dictionaries in a list, ensuring correct parsing by `data_path`. integrations/http-server/http_server/client.py [304-312] -def _extract_items_from_response(self, data: Any) -> List[Dict[str, Any]]: +def _extract_items_from_response(self, data: Any) -> Union[List[Dict[str, Any]], Dict[str, Any]]: """Extract items from API response - returns raw response for Ocean to process""" # Ocean framework will handle items_to_parse, so just return the raw response # Ocean expects the raw response and will apply items_to_parse JQ expression - if isinstance(data, list): - return data - elif isinstance(data, dict): - return [data] # Return as single item, Ocean will handle extraction - return [] + return data Apply / Chat Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a bug where wrapping a dictionary response in a list breaks the `data_path` extraction logic in `main.py`, and fixing it is crucial for handling single-item object responses.	High
Update

- Test client initialization with different auth types (none, bearer, basic, API key) - Simple, focused tests following Ocean integration patterns - All tests passing (4/4)

…ntegration

- Remove unused imports from client.py and overrides.py - Add missing query_api_for_parameters function for dynamic path parameter resolution - All linting checks passing - All tests passing (4/4)

github-actions · 2025-10-05T07:35:17Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255632040/artifacts/4185000467

Code Coverage Total Percentage: `87.18%`

- Removed HTTP Server - Slack Test configuration - Removed HTTP Server - Entra ID configuration - Keep only generic auth method examples (No Auth, Bearer Token, Basic Auth, API Key)

github-actions · 2025-10-05T07:41:16Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255691524/artifacts/4185015644

Code Coverage Total Percentage: `87.18%`

- Fixed yamllint error: too many blank lines at end of file - All lint checks now passing

- Auto-formatted 8 Python files to match black code style - All files now pass black --check validation - Ready for CI/CD pipeline

github-actions · 2025-10-05T07:47:49Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255759325/artifacts/4185030106

Code Coverage Total Percentage: `86.93%`

github-actions · 2025-10-05T07:50:22Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255784512/artifacts/4185035444

Code Coverage Total Percentage: `86.93%`

github-actions · 2025-10-05T07:50:46Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255790192/artifacts/4185036322

Code Coverage Total Percentage: `86.93%`

- Added return type annotations to functions - Fixed type hints (dict -> Dict, list -> List) - Removed unused imports (UserAgentType) - Removed unused legacy functions (query_port_entities, extract_property_value) - Added type: ignore comments where needed for Ocean framework compatibility - All tests passing (4/4) - All lint checks passing (black, ruff, mypy)

github-actions · 2025-10-05T07:57:46Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255847696/artifacts/4185049700

Code Coverage Total Percentage: `87.18%`

github-actions · 2025-10-05T08:01:14Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255872660/artifacts/4185056407

Code Coverage Total Percentage: `86.93%`

github-actions · 2025-10-05T08:06:18Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255925241/artifacts/4185070455

Code Coverage Total Percentage: `86.93%`

- Removed custom extract_data_with_jq function, now using ocean.app.integration.entity_processor._search directly - Removed backward compatibility for old selector format (only support direct fields on selector) - Removed unnecessary 'endpoint = kind' variable assignment - Cleaner, simpler code that leverages Ocean's existing functionality - All tests passing (4/4) - Zero functionality changes

github-actions · 2025-10-05T13:10:50Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18259045876/artifacts/4185836797

Code Coverage Total Percentage: `86.93%`

mk-armah

Great initiative

integrations/http-server/http_server/client.py

integrations/http-server/main.py

mk-armah · 2025-10-08T07:12:11Z

integrations/http-server/main.py

migrate resolve_dynamic_endpoints and query_api_for_parameters to a util outside of the main so they can be thoroughly tested

Fixed and added tests @mk-armah - please review

integrations/http-server/pyproject.toml

matan84

left 1 comment

integrations/http-server/http_server/client.py

- Created endpoint_resolver.py module with reusable helper functions - Extracted path parameter logic into separate functions for better testability - Removed @cache_iterator_result decorator to prevent OOM issues - Simplified main.py by moving 105 lines to dedicated module - Updated launch.json with correct example paths and merged conflicts - Maintained backward compatibility and all existing functionality

github-actions · 2025-10-10T12:39:39Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18406433848/artifacts/4237069698

Code Coverage Total Percentage: `87.52%`

github-actions · 2025-10-10T12:43:18Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18406518137/artifacts/4237099716

Code Coverage Total Percentage: `87.28%`

@patch

- Moved endpoint_resolver.py to helpers/ directory (Ocean standard pattern) - Created helpers/__init__.py with proper exports - Added 21 comprehensive tests for endpoint resolver functions - All 25 tests passing (4 client + 21 endpoint resolver) - Tests follow Ocean's mocking patterns using @patch decorator - Test coverage includes: * extract_path_parameters (4 tests) * validate_endpoint_parameters (4 tests) * generate_resolved_endpoints (4 tests) * query_api_for_parameters (3 tests with mocking) * resolve_dynamic_endpoints (6 integration tests) - All linting passes (ruff, black, mypy) - Follows same structure as GitLab, Okta, ArmorCode integrations

- Fixed yamllint errors in slack-integration.yaml - Fixed yamllint errors in dynamic-endpoints.yaml - Removed trailing blank lines at end of files

github-actions · 2025-10-10T13:09:31Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18407149482/artifacts/4237327292

Code Coverage Total Percentage: `87.34%`

github-actions · 2025-10-10T13:17:41Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18407328592/artifacts/4237407951

Code Coverage Total Percentage: `87.34%`

matan84

final comment

matan84 · 2025-10-12T10:14:46Z

integrations/http-server/http_server/client.py

+        if pagination_type == "none":
+            # Single request, no pagination
+            async for batch in self._fetch_single_page(
+                url, method, params, request_headers
+            ):
+                yield batch
+
+        elif pagination_type == "offset":
+            async for batch in self._fetch_offset_paginated(
+                url, method, params, request_headers
+            ):
+                yield batch
+
+        elif pagination_type == "page":
+            async for batch in self._fetch_page_paginated(
+                url, method, params, request_headers
+            ):
+                yield batch
+
+        elif pagination_type == "cursor":
+            async for batch in self._fetch_cursor_paginated(
+                url, method, params, request_headers
+            ):
+                yield batch


Can we make this abit more OOP friendly? Maybe have classes instead of these ifs?

Yeah good point

@matan84 - I added a dictionary, I found it quite hard to do it full OOP without breaking the ocean functions themselves.. So take a look an tell me what you think

In addition- Reset the default resources and added saas support as discussed

Major improvements to the HTTP Server integration: ## Code Quality Improvements - Refactored pagination logic to use dictionary dispatch pattern instead of if/elif chains - Removed verbose comments for cleaner, more maintainable code - Implemented OOP-friendly handler registration in __init__ - Improved pagination handling for both dict and array API responses ## Configuration Updates - Added SAAS support with recommendedInstallationMethod in spec.yaml - Enabled SAAS deployment with liveEvents disabled (generic integration) - Added empty blueprints.json to match other integrations' structure - Cleared default resources in port-app-config.yml for user configuration - Removed EntraID example files (not relevant for generic integration) ## Data Path Support - Enhanced overrides.py to properly support data_path field in selectors - Added allow_population_by_field_name for Pydantic field compatibility - Changed to use HttpServerResourceConfig for proper custom field handling ## Miscellaneous - Added .DS_Store to .gitignore for macOS users Net change: -528 lines (cleaner, more maintainable code)

github-actions · 2025-10-12T13:05:39Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18444192537/artifacts/4248192878

Code Coverage Total Percentage: `87.39%`

github-actions · 2025-10-12T13:12:48Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18444264353/artifacts/4248214544

Code Coverage Total Percentage: `87.39%`

…hitecture - Add dynamic endpoint resolution with path parameters - Support for nested resources (e.g., /tickets/{ticket_id}/comments) - Inject resolved parameters as __<param_name> in entity context - Update endpoint_resolver to return (url, params) tuples - Refactor authentication and pagination to modular handlers - Create unified handlers.py with strategy pattern - Support bearer, API key, basic auth, and none - Support none, page, offset, and cursor pagination - Use dictionary dispatch instead of if/elif chains - Simplify pagination configuration - Consolidate to paginationParam and sizeParam - Add cursorPath and hasMorePath for cursor pagination - Smart defaults based on pagination type - Add comprehensive Zendesk example - Complete blueprints for organizations, users, tickets, comments - Port configuration with path parameters - Installation examples for Docker, Helm, K8s, Docker Compose - Demonstrates cursor pagination and data_path extraction - Type safety improvements - Add comprehensive type hints to handlers - Fix MyPy compatibility for AsyncGenerator - Pass all lint, format, and type checks

github-actions · 2025-10-13T06:10:09Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18456508744/artifacts/4251469187

Code Coverage Total Percentage: `87.35%`

- Add data_path field to ApiPathParameter model - Support extracting arrays from nested responses in path parameter discovery - Fix query_api_for_parameters to collect ALL values across all pages - Update Zendesk example to use data_path in path_parameters Fixes issue where path parameters only worked with direct arrays. Now supports nested responses like {"tickets": [...], "meta": {...}}

matan84

1 final_last_v2 comment

integrations/http-server/pyproject.toml

Update endpoint resolver tests to expect (url, params) tuples instead of just URLs after path parameter injection changes

github-actions · 2025-10-13T07:23:34Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18457934111/artifacts/4251911362

Code Coverage Total Percentage: `87.33%`

Co-authored-by: Matan <[email protected]>

github-actions · 2025-10-13T07:34:26Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18458179112/artifacts/4251985165

Code Coverage Total Percentage: `87.33%`

Fix yamllint errors by removing extra blank lines at EOF

github-actions · 2025-10-13T07:37:39Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18458258905/artifacts/4252007968

Code Coverage Total Percentage: `87.33%`

github-actions · 2025-10-13T07:39:13Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18458289212/artifacts/4252018870

Code Coverage Total Percentage: `87.33%`

github-actions · 2025-10-13T07:49:34Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18458526953/artifacts/4252091062

Code Coverage Total Percentage: `87.33%`

matan84

LGTM

integrations/http-server/CHANGELOG.md

Co-authored-by: Matan <[email protected]>

github-actions · 2025-10-13T08:11:42Z

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18459004332/artifacts/4252256475

Code Coverage Total Percentage: `87.33%`

mk-armah

Approved with few suggestions on comments

mk-armah · 2025-10-13T08:08:06Z

integrations/http-server/http_server/handlers.py

+# ============================================================================
+# Authentication Handlers
+# ============================================================================


Suggested change

# ============================================================================

# Authentication Handlers

# ============================================================================

mk-armah · 2025-10-13T08:08:31Z

integrations/http-server/http_server/handlers.py

+
+# ============================================================================
+# Pagination Handlers
+# ============================================================================
+


Suggested change

# ============================================================================

# Pagination Handlers

# ============================================================================

mk-armah · 2025-10-13T08:09:06Z

integrations/http-server/http_server/handlers.py

+    ) -> AsyncGenType[List[Dict[str, Any]], None]:
+        response = await self.make_request(url, method, params, headers)
+        response_data = response.json()
+        # Yield raw response as single-item batch for Ocean's data_path extraction


Suggested change

# Yield raw response as single-item batch for Ocean's data_path extraction

mk-armah · 2025-10-13T08:11:59Z

integrations/http-server/http_server/client.py

+        self.timeout = timeout
+        self.verify_ssl = verify_ssl
+
+        # Use Ocean's built-in HTTP client with retry and rate limiting


Suggested change

# Use Ocean's built-in HTTP client with retry and rate limiting

mk-armah · 2025-10-13T08:12:19Z

integrations/http-server/http_server/client.py

+        auth_handler = get_auth_handler(self.auth_type, self.client, self.auth_config)
+        auth_handler.setup()
+
+        # Concurrency control


Suggested change

# Concurrency control

Copilot AI review requested due to automatic review settings October 5, 2025 07:21

github-actions bot added the size/XXL label Oct 5, 2025

Copilot AI reviewed Oct 5, 2025

View reviewed changes

qodo-merge-pro bot added the Review effort 4/5 label Oct 5, 2025

Matarp13 added 3 commits October 5, 2025 10:26

test: Add basic client tests for HTTP Server integration

09bf598

- Test client initialization with different auth types (none, bearer, basic, API key) - Simple, focused tests following Ocean integration patterns - All tests passing (4/4)

Merge remote-tracking branch 'origin/main' into feature/http-server-i…

6211e36

…ntegration

fix: Resolve linting issues in HTTP Server integration

e933d72

- Remove unused imports from client.py and overrides.py - Add missing query_api_for_parameters function for dynamic path parameter resolution - All linting checks passing - All tests passing (4/4)

Update pyproject.toml

4b02e06

matarpeles changed the title ~~feat: Add HTTP Server integration with dynamic endpoints and AI wizar…~~ feat: Add HTTP Server integration with dynamic endpoints Oct 5, 2025

chore: Remove Slack and Entra ID test configurations from launch.json

055a481

- Removed HTTP Server - Slack Test configuration - Removed HTTP Server - Entra ID configuration - Keep only generic auth method examples (No Auth, Bearer Token, Basic Auth, API Key)

Matarp13 added 3 commits October 5, 2025 10:42

chore: Update poetry.lock to match pyproject.toml changes

333578e

fix: Remove extra blank lines from entraid-blueprints.json

5156f6f

- Fixed yamllint error: too many blank lines at end of file - All lint checks now passing

style: Format HTTP server integration with black

8db9780

- Auto-formatted 8 Python files to match black code style - All files now pass black --check validation - Ready for CI/CD pipeline

mk-armah requested changes Oct 8, 2025

View reviewed changes

matan84 requested changes Oct 9, 2025

View reviewed changes

integrations/http-server/http_server/client.py Outdated Show resolved Hide resolved

Matarp13 added 2 commits October 10, 2025 15:55

fix(http-server): remove extra blank lines in yaml examples

e619925

- Fixed yamllint errors in slack-integration.yaml - Fixed yamllint errors in dynamic-endpoints.yaml - Removed trailing blank lines at end of files

matan84 requested changes Oct 12, 2025

View reviewed changes

Matarp13 added 2 commits October 12, 2025 15:51

fix(http-server): remove unused imports from overrides.py

3f67e25

matan84 requested changes Oct 13, 2025

View reviewed changes

integrations/http-server/pyproject.toml Outdated Show resolved Hide resolved

test(http-server): Update tests for tuple return type

239acbd

Update endpoint resolver tests to expect (url, params) tuples instead of just URLs after path parameter injection changes

matarpeles and others added 2 commits October 13, 2025 10:23

Update .gitignore

4753131

Update integrations/http-server/pyproject.toml

6217978

Co-authored-by: Matan <[email protected]>

fix(http-server): Remove trailing blank lines from Zendesk examples

2a361b9

Fix yamllint errors by removing extra blank lines at EOF

matan84 approved these changes Oct 13, 2025

View reviewed changes

integrations/http-server/CHANGELOG.md Outdated Show resolved Hide resolved

Update integrations/http-server/CHANGELOG.md

c638553

Co-authored-by: Matan <[email protected]>

mk-armah approved these changes Oct 13, 2025

View reviewed changes

matarpeles merged commit 1c57215 into main Oct 13, 2025
30 checks passed

matarpeles deleted the feature/http-server-integration branch October 13, 2025 08:25

		field: str = Field(description="JQ expression to extract parameter value from each record")
		filter: Optional[str] = Field(default=None, description="JQ boolean expression to filter records")

		search_query = param_config.search
		property_path = param_config.property

	# ============================================================================
	# Authentication Handlers
	# ============================================================================

feat: Add HTTP Server integration with dynamic endpoints #2241

feat: Add HTTP Server integration with dynamic endpoints #2241

Uh oh!

Conversation

matarpeles commented Oct 5, 2025 • edited by qodo-merge-pro bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

User description

Description

Type of change

All tests should be run against the port production environment(using a testing org).

Core testing checklist

Integration testing checklist

Preflight checklist

Screenshots

API Documentation

PR Type

Description

Diagram Walkthrough

File Walkthrough

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 5, 2025

Choose a reason for hiding this comment

Uh oh!

qodo-merge-pro bot commented Oct 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Compliance Guide 🔍

Uh oh!

qodo-merge-pro bot commented Oct 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Code Suggestions ✨

Examples:

Solution Walkthrough:

Before:

After:

Uh oh!

github-actions bot commented Oct 5, 2025

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255632040/artifacts/4185000467

Code Coverage Total Percentage: 87.18%

Uh oh!

github-actions bot commented Oct 5, 2025

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255691524/artifacts/4185015644

Code Coverage Total Percentage: 87.18%

Uh oh!

github-actions bot commented Oct 5, 2025

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255759325/artifacts/4185030106

Code Coverage Total Percentage: 86.93%

Uh oh!

github-actions bot commented Oct 5, 2025

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255784512/artifacts/4185035444

Code Coverage Total Percentage: 86.93%

Uh oh!

github-actions bot commented Oct 5, 2025

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255790192/artifacts/4185036322

Code Coverage Total Percentage: 86.93%

Uh oh!

github-actions bot commented Oct 5, 2025

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255847696/artifacts/4185049700

Code Coverage Total Percentage: 87.18%

Uh oh!

github-actions bot commented Oct 5, 2025

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255872660/artifacts/4185056407

Code Coverage Total Percentage: 86.93%

Uh oh!

github-actions bot commented Oct 5, 2025

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18255925241/artifacts/4185070455

Code Coverage Total Percentage: 86.93%

Uh oh!

github-actions bot commented Oct 5, 2025

Code Coverage Artifact 📈: https://github.com/port-labs/ocean/actions/runs/18259045876/artifacts/4185836797

Code Coverage Total Percentage: 86.93%

Uh oh!

mk-armah left a comment

matarpeles commented Oct 5, 2025 •

edited by qodo-merge-pro bot

Loading

qodo-merge-pro bot commented Oct 5, 2025 •

edited

Loading

qodo-merge-pro bot commented Oct 5, 2025 •

edited

Loading

Code Coverage Total Percentage: `87.18%`

Code Coverage Total Percentage: `87.18%`

Code Coverage Total Percentage: `86.93%`

Code Coverage Total Percentage: `86.93%`

Code Coverage Total Percentage: `86.93%`

Code Coverage Total Percentage: `87.18%`

Code Coverage Total Percentage: `86.93%`

Code Coverage Total Percentage: `86.93%`

Code Coverage Total Percentage: `86.93%`

Code Coverage Total Percentage: `87.52%`

Code Coverage Total Percentage: `87.28%`

Code Coverage Total Percentage: `87.34%`

Code Coverage Total Percentage: `87.34%`

Code Coverage Total Percentage: `87.39%`

Code Coverage Total Percentage: `87.39%`

Code Coverage Total Percentage: `87.35%`