Skip to content

Conversation

aaronsteers
Copy link
Contributor

feat: add get_connector_builder_docs tool for LLM guidance

Summary

This PR adds a new get_connector_builder_docs() function to the builder-mcp repository that provides documentation guidance for Airbyte connector development. The function serves as a comprehensive documentation tool for LLM agents building low-code YAML-based connectors.

Key Features:

  • High-level overview: Returns step-by-step process guidance and available tools when no specific topic is requested
  • Topic-specific documentation: Fetches detailed documentation for specific topics using raw GitHub URLs
  • Comprehensive topic mapping: Covers both Connector Builder UI and YAML reference documentation from the airbyte repository
  • FastMCP integration: Follows existing tool registration patterns for seamless integration

Implementation Details:

  • Added main function get_connector_builder_docs() with optional topic parameter
  • Implemented _get_high_level_overview() with process steps and tool descriptions
  • Implemented _get_topic_specific_docs() with GitHub raw URL fetching
  • Added requests dependency and types-requests for type checking
  • Registered the new tool with FastMCP following existing patterns

Review & Testing Checklist for Human

  • Test the function end-to-end: Call get_connector_builder_docs() with no parameters to verify the high-level overview is returned correctly
  • Test topic-specific functionality: Call the function with various topic parameters (e.g., "authentication", "pagination") to verify documentation is fetched from GitHub
  • Verify topic mapping accuracy: Cross-check the hardcoded topic mapping against actual files in the GitHub directories to ensure paths are correct and complete
  • Test error handling: Verify graceful handling when network is unavailable or when requesting non-existent topics
  • Confirm MCP integration: Test that the new tool is properly registered and accessible through the MCP server interface

Recommended Test Plan: Start the MCP server locally and use an MCP client to call the new tool with both overview and specific topic requests to verify full functionality.


Diagram

%%{ init : { "theme" : "default" }}%%
graph TD
    Server["builder_mcp/server.py"]:::context
    ConnectorBuilder["builder_mcp/_connector_builder.py"]:::major-edit
    PyProject["pyproject.toml"]:::minor-edit
    GitHubDocs["GitHub Raw URLs<br/>(airbytehq/airbyte/docs)"]:::context
    
    Server -->|"registers tools"| ConnectorBuilder
    ConnectorBuilder -->|"fetches docs via HTTP"| GitHubDocs
    PyProject -->|"provides requests dependency"| ConnectorBuilder
    
    subgraph "New Functions"
        GetDocs["get_connector_builder_docs()"]:::major-edit
        HighLevel["_get_high_level_overview()"]:::major-edit  
        TopicSpecific["_get_topic_specific_docs()"]:::major-edit
    end
    
    ConnectorBuilder --> GetDocs
    GetDocs --> HighLevel
    GetDocs --> TopicSpecific
    TopicSpecific --> GitHubDocs
    
    subgraph Legend
        L1["Major Edit"]:::major-edit
        L2["Minor Edit"]:::minor-edit
        L3["Context/No Edit"]:::context
    end
    
    classDef major-edit fill:#90EE90
    classDef minor-edit fill:#87CEEB  
    classDef context fill:#FFFFFF
Loading

Notes

  • The topic mapping includes 25+ documentation topics covering both Connector Builder UI and YAML reference documentation
  • Function uses raw GitHub URLs for direct access to markdown content, avoiding HTML parsing complexity
  • Error handling includes fallback to docs.airbyte.com URLs when GitHub fetching fails
  • Implementation maintains consistency with existing MCP tool patterns in the codebase

Session Details:

- Add comprehensive connector building documentation function
- Provide high-level process guidance and available tools overview
- Support topic-specific documentation via raw GitHub URLs
- Follow existing FastMCP tool registration patterns
- Add requests dependency and types-requests for type checking

Requested by @aaronsteers for builder-mcp repository

Co-Authored-By: AJ Steers <[email protected]>
Copy link
Contributor

Original prompt from AJ Steers
@Devin - Let's add docs and guidance to the LLM tools in the builder-mcp repo. We probably need a "get_connector_builder_docs()" that returns high-level instructions if no specific topic is requested. It can also return a list of available topics. We should have a focused result for every subpage of this: <https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/reference>

@Copilot Copilot AI review requested due to automatic review settings August 1, 2025 21:04
Copy link
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions github-actions bot added the enhancement New feature or request label Aug 1, 2025
Copy link

github-actions bot commented Aug 1, 2025

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Testing This Version

You can test this version of the MCP Server using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/builder-mcp.git@devin/1754081872-add-connector-builder-docs#egg=airbyte-builder-mcp' --help

Helpful Resources

  • TBA

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /poe <command> - Runs any poe command in the uv virtual environment

📝 Edit this welcome message.

Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a new get_connector_builder_docs() function that provides comprehensive documentation guidance for Airbyte connector development, serving as a documentation tool for LLM agents building low-code YAML-based connectors.

  • Implements a documentation retrieval system with high-level overview and topic-specific content
  • Adds comprehensive topic mapping covering 25+ Connector Builder UI and YAML reference topics
  • Integrates with existing FastMCP tool registration patterns for seamless operation

Reviewed Changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.

File Description
pyproject.toml Adds requests dependency and types-requests for HTTP documentation fetching
builder_mcp/_connector_builder.py Implements new documentation tool with overview, topic-specific docs, and MCP registration

Copy link

github-actions bot commented Aug 1, 2025

PyTest Results (Fast)

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit d32981b. ± Comparison against base commit 637fe2f.

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Aug 1, 2025

PyTest Results (Full)

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit d32981b. ± Comparison against base commit 637fe2f.

♻️ This comment has been updated with latest results.

- Remove duplicate [project.optional-dependencies] section
- Consolidate all dev dependencies in [dependency-groups] section
- Use higher version constraints to avoid conflicts
- Fixes 'Build and Inspect Python Package' CI check failure

Co-Authored-By: AJ Steers <[email protected]>
@aaronsteers
Copy link
Contributor Author

I updated from main to resolve the unrelated ci failure.

- Remove redundant file_ext parameter from topic mapping tuples
- Update tuple structure from (relative_path, file_ext, description) to (relative_path, description)
- Derive file extension from relative_path when needed for docs URL construction
- Addresses GitHub comment feedback about removing duplication
- Maintains all existing functionality while making code more DRY

Co-Authored-By: AJ Steers <[email protected]>
aaronsteers and others added 3 commits August 1, 2025 14:46
…sible

- Add TestConnectorBuilderDocs class with parametrized test for all topics
- Test verifies HTTP 200 status code for each topic's GitHub raw URL
- Helps identify any broken links in the topic mapping
- Addresses GitHub comment feedback from @aaronsteers

Co-Authored-By: AJ Steers <[email protected]>
@aaronsteers
Copy link
Contributor Author

aaronsteers commented Aug 1, 2025

/autofix

Auto-Fix Job Info

This job attempts to auto-fix any linting or formating issues. If any fixes are made,
those changes will be automatically committed and pushed back to the PR.

Note: This job can only be run by maintainers. On PRs from forks, this command requires
that the PR author has enabled the Allow edits from maintainers option.

PR auto-fix job started... Check job output.

✅ Changes applied successfully.

@aaronsteers aaronsteers merged commit cb55b1d into main Aug 1, 2025
13 checks passed
@aaronsteers aaronsteers deleted the devin/1754081872-add-connector-builder-docs branch August 1, 2025 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant