Custom components use case analysis: HTTP Requests for Configuration Determination #713
Description
This issue documents a use case That requires custom Python. The purpose of this issue is to document the requirement and make a space for discussing if it can be supported in a YAML only format.
Use Case
Some manifest-only connectors need to make HTTP requests during configuration setup to determine correct API endpoints or extract configuration values before stream processing begins.
Example Implementation
source-mailchimp uses ExtractAndSetDataCenterConfigValue to make OAuth metadata requests and extract the data center from authentication credentials:
- Makes HTTP request to OAuth token info endpoint
- Extracts data center from response
- Modifies connector configuration with correct API base URL
- Handles both OAuth and API key authentication methods
Code Reference:
https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mailchimp/components.py#L44-L93
Current Gap
The YAML dialect cannot perform runtime HTTP requests to modify configuration during connector initialization, requiring custom Python components for this functionality.