Survey of Manifest-Only Connectors Using Custom Components

[devin-ai-integration]

Analysis of 5 manifest-only connectors that use custom components.py files:

Connectors Analyzed

source-gitlab (certified)

• Custom URL validation with environment-specific logic
• Complex partition routing for hierarchical data relationships
• Code: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-gitlab/components.py

source-mailchimp (certified)

• Runtime HTTP requests for data center extraction: Custom components use case analysis: HTTP Requests for Configuration Determination #713
• Custom record extraction for flattening nested arrays
• Code: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mailchimp/components.py

source-google-sheets (certified)

• Complex schema parsing and name conversion with configurable sanitization
• Dynamic range-based partitioning for batch processing
• Code: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-sheets/components.py

source-younium (community)

• Custom OAuth2 authenticator with environment-specific endpoints
• Token generation with conditional URL selection
• Code: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-younium/components.py

source-klaviyo (certified)

• Custom record transformations with additional API calls
• Sophisticated relationship data merging from API responses
• Custom error handling for DNS resolution issues
• Code: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-klaviyo/components.py

Summary

All analyzed connectors implement functionality requiring custom Python logic beyond current YAML dialect capabilities.

[katmarkham]

@aaronsteers why did you select these 5 connectors?

[aaronsteers]

@katmarkham - just a random sampling.

[aaronsteers]

Now having thought on this more... it could be interesting to analyze just the certified connectors. That was not a requirement when I ran the the first time though

[devin-ai-integration]

Certified Manifest-Only Connectors: Custom Components Usage

Summary Statistics

• Total Certified Manifest-Only Connectors: 31
• Using Custom Components: 23 (74.2%)
• Pure YAML Implementation: 8 (25.8%)

Certified Manifest-Only Connectors WITHOUT Custom Components (8 connectors)

1. source-airtable
2. source-amazon-ads
3. source-harvest
4. source-sendgrid
5. source-sentry
6. source-snapchat-marketing
7. source-stripe
8. source-woocommerce

Certified Manifest-Only Connectors WITH Custom Components (23 connectors)

Authentication & Configuration

• source-younium - Custom OAuth2 authenticator with environment-specific endpoints (sandbox vs production)
  • Components
• source-mailchimp - Runtime HTTP requests to extract data center from OAuth metadata and modify API base URL
  • Components
• source-gitlab - Environment-aware URL validation (HTTP vs HTTPS based on cloud environment)
  • Components

Complex Data Processing & Transformation

• source-amazon-seller-partner - Multiple custom decoders for GZIP CSV/XML/JSON, marketplace-specific date formatting, custom validation
  • Components
• source-bing-ads - Complex campaign record transformations, custom CSV decoders, duplicate record filtering, state migration
  • Components
• source-hubspot - Dynamic schema loading, property history extraction, legacy field transformations, endpoint-specific routing
  • Components
• source-instagram - Complex media transformations, insights flattening, URL parameter cleaning, additional API calls for enrichment
  • Components
• source-google-sheets - Schema parsing with configurable sanitization, dynamic range partitioning, name conversion logic
  • Components
• source-klaviyo - Relationship data merging, campaign enrichment via additional API calls, custom error handling
  • Components

Advanced Pagination & Routing

• source-freshdesk - Custom pagination with cursor-based continuation when page limits are hit
  • Components
• source-intercom - Custom rate limiting, scroll pagination with reset handling, error recovery logic
  • Components
• source-gitlab - Dynamic partition routing based on group/project hierarchies
  • Components

Custom Record Extraction & Schema Generation

• source-google-analytics-data-api - Combined extractors for merging multiple data sources, key-value mapping from flat arrays
  • Components
• source-google-search-console - Dynamic schema generation based on selected dimensions, custom state migration for nested substreams
  • Components
• source-amplitude - Complex response structure parsing for analytics data, datetime format transformations
  • Components

Simple Field Transformations

• source-chargebee - Custom field detection and transformation (cf_ prefix handling)
  • Components
• source-jira - Field mapping and data structure normalization
  • Components
• source-linkedin-ads - Custom transformations for advertising data formats
  • Components
• source-monday - Workspace and board data transformations
  • Components
• source-paypal-transaction - Transaction data normalization and formatting
  • Components
• source-tiktok-marketing - Marketing campaign data transformations
  • Components
• source-typeform - Form response data processing
  • Components
• source-zendesk-chat - Chat data formatting and normalization
  • Components
• source-zendesk-support - Support ticket data transformations
  • Components
• source-zendesk-talk - Call center data processing
  • Components

[devin-ai-integration]

Community Manifest-Only Connectors Analysis

Summary Statistics

• Total Community Manifest-Only Connectors: 448
• Using Custom Components: 28 (6.3%)
• Pure YAML Implementation: 420 (93.7%)

Community Manifest-Only Connectors WITH Custom Components (28 connectors)

1. source-alpha-vantage - Components
2. source-apify-dataset - Components
3. source-aws-cloudtrail - Components
4. source-bamboo-hr - Components
5. source-braze - Components
6. source-fastbill - Components
7. source-gnews - Components
8. source-gong - Components
9. source-greenhouse - Components
10. source-instatus - Components
11. source-my-hours - Components
12. source-orb - Components
13. source-outreach - Components
14. source-pipedrive - Components
15. source-pocket - Components
16. source-prestashop - Components
17. source-qualaroo - Components
18. source-railz - Components
19. source-recurly - Components
20. source-retently - Components
21. source-salesloft - Components
22. source-the-guardian-api - Components
23. source-trello - Components
24. source-us-census - Components
25. source-xero - Components
26. source-younium - Components
27. source-zendesk-sunshine - Components
28. source-zoom - Components

Community Manifest-Only Connectors WITHOUT Custom Components (420 connectors)

Pure YAML implementation connectors include source-openfda, source-pennylane, source-humanitix, source-picqer, and 416 others.