feat: Add bin/index_build.py script to create searchable connector component index #26

aaronsteers · 2025-08-08T21:47:10Z

feat: Switch connector feature index to CSV output with improved filtering

Summary

This PR updates the connector feature index script with significant changes to output format and filtering logic based on user feedback:

Output format change: Switched from JSON to CSV format with "FeatureUsage" and "ConnectorName" columns
Improved filtering: Added requirement for at least one lowercase character to exclude all-caps matches like 'A13V1IB3VIYZZH'
Script rename: bin/index_build.py → bin/build_connector_feature_index.py
Output location: connector_component_index.json → generated/connector-feature-index.csv
Added poe task: New "build" task in pyproject.toml to invoke the script

The script processes 478 manifest.yaml files and now generates 14,010 feature-connector pairs (previously 2,271 unique class names). The CSV format is sorted by feature name first, then connector name, making it easier to scan and maintain.

Review & Testing Checklist for Human

End-to-end testing: Run poe build and verify it completes without errors and generates the CSV file
CSV output verification: Open generated/connector-feature-index.csv and spot-check that the data looks correct with proper headers and sorting
Filtering validation: Manually verify that all-caps identifiers (like Amazon marketplace IDs) are excluded while legitimate class names like "DeclarativeSource" are included
Class name extraction accuracy: Sample a few manifest.yaml files and manually verify that the regex patterns are correctly extracting class names without major false positives/negatives
Data integrity: Compare a subset of results with the previous JSON output to ensure no legitimate class names were lost in the format conversion

Recommended test plan: Run the script, examine the CSV output structure, and manually verify 5-10 connectors' extracted class names against their actual manifest.yaml files.

Diagram

%%{ init : { "theme" : "default" }}%%
graph TD
    PyProject["pyproject.toml<br/>poe build task"]:::minor-edit --> Script
    Script["bin/build_connector_feature_index.py<br/>Main script logic"]:::major-edit
    Script --> TempRepo["Temporary airbyte repo<br/>Shallow clone"]:::context
    TempRepo --> Manifests["478 manifest.yaml files<br/>Source connectors"]:::context
    Manifests --> Extraction["Class name extraction<br/>Regex + filtering"]:::major-edit
    Extraction --> CSVOutput["generated/connector-feature-index.csv<br/>14,010 rows"]:::major-edit
    
    
    subgraph Legend
        L1[Major Edit]:::major-edit
        L2[Minor Edit]:::minor-edit  
        L3[Context/No Edit]:::context
    end

    classDef major-edit fill:#90EE90
    classDef minor-edit fill:#87CEEB
    classDef context fill:#FFFFFF

Notes

The filtering change reduced unique class names from 2,271 to 2,084, indicating the lowercase requirement successfully filtered out unwanted all-caps matches
CI checks mostly passed (10/11) with only docs preview failing (likely unrelated)
The CSV format creates a flattened structure that may be easier for analysis tools to consume
Session requested by: AJ Steers (@aaronsteers)
Devin session: https://app.devin.ai/sessions/dbb4d0c14cf34055b67b00dfa8f5386c

…mponent index - Creates searchable index mapping class names to connectors that use them - Shallow-checkouts airbytehq/airbyte repo to temp directory - Scans all manifest.yaml files in airbyte-integrations/connectors/source-*/ - Extracts ClassName-formatted identifiers using regex patterns - Filters out common false positives (HTTP methods, acronyms, etc.) - Generates JSON index with 2,271+ unique class names from 478+ connectors - Provides summary statistics and usage examples - Enables discovery of connectors using specific features/components Co-Authored-By: AJ Steers <[email protected]>

devin-ai-integration · 2025-08-08T21:47:13Z

Original prompt from AJ Steers

@Devin - For the connector builder MCP project, could you create an bin/index_build.py script that does the following:
1. Create a list of all class names (cased like ClassName) in the manifest connector json schema yaml file. (In the CDK master branch. See tool that gets this already in the builder.)
2. Shallow-checkout the airgbytehq/airbyte repo to a temp directory.
3. Create a word cloud dataset of all class names (cased like ClassName) within all manifest.yaml files in the airbyte repo, specifically if they exactly match the glob: airbyte-integrations/connectors/source-{something}/manifest.yaml
4. We want a mapping of all class names possible - mapped to all connectors that use them.
An alternative approach is to skip step 1 and just get presumed class names from regex against the manifest.yaml files - this might be easier actually.

Extra results in the index that we don't end up using won't hurt us. Basically we're creating a searchable index of connectors using specific features or components.

Copilot

Copilot wasn't able to review any files in this pull request.

devin-ai-integration · 2025-08-08T21:47:14Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment and CI monitoring

github-actions · 2025-08-08T21:47:23Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Testing This Branch via MCP

To test the changes in this specific branch with an MCP client like Claude Desktop, use the following configuration:

{
  "mcpServers": {
    "connector-builder-mcp-dev": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/airbytehq/connector-builder-mcp.git@devin/1723158344-index-build-script", "connector-builder-mcp"]
    }
  }
}

Testing This Branch via CLI

You can test this version of the MCP Server using the following CLI snippet:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/connector-builder-mcp.git@devin/1723158344-index-build-script#egg=airbyte-connector-builder-mcp' --help

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

/autofix - Fixes most formatting and linting issues
/poe <command> - Runs any poe command in the uv virtual environment

📝 Edit this welcome message.

github-actions · 2025-08-08T21:48:14Z

PyTest Results (Fast)

0 tests ±0 0 ✅ ±0 0s ⏱️ ±0s
0 suites ±0 0 💤 ±0
0 files ±0 0 ❌ ±0

Results for commit 82ff059. ± Comparison against base commit 620ad19.

♻️ This comment has been updated with latest results.

- Rename bin/index_build.py to bin/build_connector_feature_index.py - Update output path from connector_component_index.json to generated/connector-feature-index.json - Add poe task 'build' to invoke the script - Create generated/ directory for output files - Verify script works correctly with new configuration Co-Authored-By: AJ Steers <[email protected]>

Co-Authored-By: AJ Steers <[email protected]>

github-actions · 2025-08-09T00:11:24Z

PyTest Results (Full)

0 tests ±0 0 ✅ ±0 0s ⏱️ ±0s
0 suites ±0 0 💤 ±0
0 files ±0 0 ❌ ±0

Results for commit 82ff059. ± Comparison against base commit 620ad19.

♻️ This comment has been updated with latest results.

- Add lowercase character requirement to exclude all-caps matches like 'A13V1IB3VIYZZH' - Switch from JSON to CSV format with 'FeatureUsage' and 'ConnectorName' columns - Sort output by feature name first, then by connector name - Update output filename to connector-feature-index.csv - Reduce unique class names from 2,271 to 2,084 with better filtering Co-Authored-By: AJ Steers <[email protected]>

aaronsteers · 2025-08-09T00:45:16Z

bin/build_connector_feature_index.py

+        class_names = extract_class_names_from_yaml(yaml_content)
+
+        filtered_class_names = set()
+        false_positives = {


Devin, just dynamically exclude any keywords that are ALLCAPS or alllower.

@aaronsteers

…iltering - Remove hardcoded false_positives set with specific keywords - Use dynamic filtering to exclude ALLCAPS and alllower strings - Keep mixed-case class names like 'DeclarativeSource' - Simplifies maintenance and improves filtering robustness Addresses GitHub comment from @aaronsteers Co-Authored-By: AJ Steers <[email protected]>

…ucture - Move CSV output to connector_builder_mcp/resources/generated/ - Add find_connectors_by_feature() MCP tool for exact feature matching - Tool accepts comma-separated features and returns connectors with ALL features - Update build script to output to new location - Remove old JSON output file - Add csv import for proper CSV file handling Co-Authored-By: AJ Steers <[email protected]>

- Delete generated/connector-feature-index.json (replaced by CSV) - Delete generated/connector-feature-index.csv (moved to connector_builder_mcp/resources/generated/) Co-Authored-By: AJ Steers <[email protected]>

Co-Authored-By: AJ Steers <[email protected]>

aaronsteers · 2025-08-09T02:01:31Z

pyproject.toml

 [tool.deptry]
 ignore = ["DEP002"]
+
+[tool.poe.tasks]


The Poe commands should be in the dedicated tasks file

@aaronsteers

- Move build task from pyproject.toml to poe_tasks.toml per @aaronsteers feedback - Verified task still works correctly with 'poe build' - Addresses GitHub PR comment about using dedicated tasks file Co-Authored-By: AJ Steers <[email protected]>

Copilot AI review requested due to automatic review settings August 8, 2025 21:47

devin-ai-integration bot assigned aaronsteers Aug 8, 2025

Copilot AI reviewed Aug 8, 2025

View reviewed changes

github-actions bot added the enhancement New feature or request label Aug 8, 2025

devin-ai-integration bot and others added 2 commits August 9, 2025 00:10

style: Apply ruff formatting to renamed script

2fe5315

Co-Authored-By: AJ Steers <[email protected]>

aaronsteers commented Aug 9, 2025

View reviewed changes

devin-ai-integration bot and others added 4 commits August 9, 2025 00:46

chore: Remove old generated files after reorganization

e024a8a

- Delete generated/connector-feature-index.json (replaced by CSV) - Delete generated/connector-feature-index.csv (moved to connector_builder_mcp/resources/generated/) Co-Authored-By: AJ Steers <[email protected]>

fix: Add type annotation for feature_to_connectors to resolve MyPy error

b10f9bc

Co-Authored-By: AJ Steers <[email protected]>

aaronsteers commented Aug 9, 2025

View reviewed changes

aaronsteers merged commit 2510724 into main Aug 9, 2025
13 checks passed

aaronsteers deleted the devin/1723158344-index-build-script branch August 9, 2025 02:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add bin/index_build.py script to create searchable connector component index #26

feat: Add bin/index_build.py script to create searchable connector component index #26

Uh oh!

aaronsteers commented Aug 8, 2025 •

edited by devin-ai-integration bot

Loading

Uh oh!

devin-ai-integration bot commented Aug 8, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

devin-ai-integration bot commented Aug 8, 2025

Uh oh!

github-actions bot commented Aug 8, 2025

Uh oh!

github-actions bot commented Aug 8, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Aug 9, 2025 •

edited

Loading

Uh oh!

aaronsteers Aug 9, 2025

Uh oh!

aaronsteers Aug 9, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

feat: Add bin/index_build.py script to create searchable connector component index #26

feat: Add bin/index_build.py script to create searchable connector component index #26

Uh oh!

Conversation

aaronsteers commented Aug 8, 2025 • edited by devin-ai-integration bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

feat: Switch connector feature index to CSV output with improved filtering

Summary

Review & Testing Checklist for Human

Diagram

Notes

Uh oh!

devin-ai-integration bot commented Aug 8, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot commented Aug 8, 2025

🤖 Devin AI Engineer

Uh oh!

github-actions bot commented Aug 8, 2025

👋 Greetings, Airbyte Team Member!

Testing This Branch via MCP

Testing This Branch via CLI

PR Slash Commands

Uh oh!

github-actions bot commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PyTest Results (Fast)

Uh oh!

github-actions bot commented Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PyTest Results (Full)

Uh oh!

aaronsteers Aug 9, 2025

Choose a reason for hiding this comment

Uh oh!

aaronsteers Aug 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

aaronsteers commented Aug 8, 2025 •

edited by devin-ai-integration bot

Loading

github-actions bot commented Aug 8, 2025 •

edited

Loading

github-actions bot commented Aug 9, 2025 •

edited

Loading