feat: add mcp-use agent wrapper testing

[aaronsteers]

A simple test of the mcp-use agent framework.

Important

Auto-merge enabled.

This PR is set to merge automatically when all requirements are met.

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Testing This Branch via MCP

To test the changes in this specific branch with an MCP client like Claude Desktop, use the following configuration:

{
  "mcpServers": {
    "connector-builder-mcp-dev": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/airbytehq/connector-builder-mcp.git@aj/examples/mcp-use-script", "connector-builder-mcp"]
    }
  }
}

Testing This Branch via CLI

You can test this version of the MCP Server using the following CLI snippet:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/connector-builder-mcp.git@aj/examples/mcp-use-script#egg=airbyte-connector-builder-mcp' --help

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poe <command> - Runs any poe command in the uv virtual environment

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit 0df9fef. ± Comparison against base commit adf33bf.

♻️ This comment has been updated with latest results.

[aaronsteers]

Test script works. ✅

Show/Hide Detailed Log

AJ-Steers-MacBook-Pro---GGHWM7QWPJ:builder-mcp ajsteers$ uv run examples/run_mcp_use_demo.py
🚀 mcp-use + connector-builder-mcp Integration Demo
============================================================

This demo shows how mcp-use can wrap connector-builder-mcp
to provide vendor-neutral access to Airbyte connector development tools.

🔧 Demo 1: Direct Tool Calls
==================================================
<frozen abc>:106: ExperimentalClassWarning: This class is experimental. Use at your own risk.
ExperimentalClassWarning: This class is experimental. Use at your own risk.
Starting Builder MCP server.
[08/12/25 15:34:07] INFO     Starting MCP server 'connector-builder-mcp' with transport 'stdio'            server.py:1307
📋 Available MCP Tools:
  • validate_manifest: Validate a connector manifest structure and configuration.

Returns:
    Validation result with success status and any errors/warnings
  • execute_stream_test_read: Execute reading from a connector stream.

Return record data and/or raw request/response metadata from the stream test.
We attempt to automatically sanitize raw data to prevent exposure of secrets.
We do not attempt to sanitize record data, as it is expected to be user-defined.
  • execute_record_counts_smoke_test: Execute a smoke test to count records from all streams in the connector.

This function tests all available streams by reading records up to the specified limit
and returns comprehensive statistics including record counts, errors, and timing information.

Args:
    manifest: The connector manifest (YAML string or file path)
    config: Connector configuration
    max_records: Maximum number of records to read per stream (default: 10000)
    dotenv_path: Optional path to .env file for secret hydration

Returns:
    MultiStreamSmokeTest result with per-stream statistics and overall summary
  • execute_dynamic_manifest_resolution_test: Get the resolved connector manifest, expanded with detected dynamic streams and schemas.

This tool is helpful for discovering dynamic streams and schemas. This should not replace the
original manifest, but it can provide helpful information to understand how the manifest will
be resolved and what streams will be available at runtime.

Args:
    manifest: The connector manifest to resolve. Can be raw YAML content or path to YAML file
    config: Optional configuration for resolution

TODO:
- Research: Is there any reason to ever get the non-fully resolved manifest?

Returns:
    Resolved manifest or error message
  • get_manifest_yaml_json_schema: Retrieve the connector manifest JSON schema from the Airbyte repository.

This tool fetches the official JSON schema used to validate connector manifests.
The schema defines the structure, required fields, and validation rules for
connector YAML configurations.

Returns:
    Response containing the schema in YAML format
  • get_connector_builder_checklist: Get the comprehensive development checklist for building declarative source connectors.

This checklist provides a step-by-step guide for building connectors using the Connector Builder MCP Server,
with emphasis on proper validation, pagination testing, and avoiding common pitfalls.

Returns:
    Complete development checklist in markdown format
  • get_connector_builder_docs: Get connector builder documentation and guidance.

Args:
    topic: Optional specific topic from YAML reference documentation

Returns:
    High-level overview with topic list, or detailed topic-specific documentation
  • get_connector_manifest: Get the raw connector manifest YAML from connectors.airbyte.com.

Args:
    connector_name: Name of the connector (e.g., 'source-stripe')
    version: Version of the connector manifest to retrieve (defaults to 'latest')

Returns:
    Raw YAML content of the connector manifest
  • find_connectors_by_class_name: Find connectors that use ALL specified class names/components.

This tool searches for connectors that implement specific declarative component classes.

Examples of valid class names:
- DynamicDeclarativeStream (for dynamic stream discovery)
- HttpComponentsResolver (for HTTP-based component resolution)
- ConfigComponentsResolver (for config-based component resolution)
- OAuthAuthenticator (for OAuth authentication)
- ApiKeyAuthenticator (for API key authentication)

Args:
    class_names: Comma-separated string of exact class names to search for.
                Use class names like "DynamicDeclarativeStream", not feature
                descriptions like "dynamic streams" or "pagination".

Returns:
    List of connector names that use ALL specified class names
  • list_dotenv_secrets: List all secrets in the specified dotenv file without exposing values.

Args:
    dotenv_path: Path to the .env file to list secrets from

Returns:
    Information about the secrets file and its contents
  • populate_dotenv_missing_secrets_stubs: Add secret stubs to the specified dotenv file for the user to fill in.

Supports two modes:
1. Manifest-based: Pass manifest to auto-detect secrets from connection_specification
2. Path-based: Pass config_paths list like 'credentials.password,oauth.client_secret'

If both are provided, both sets of secrets will be added.

This function is non-destructive and will not overwrite existing secrets.
If any of the secrets to be added already exist, an error will be returned
with information about the existing secrets.

Returns:
    Message about the operation result

✅ Found 11 tools available

🔍 Validating sample manifest...
📄 Validation Result:
  {
  "is_valid": false,
  "errors": [
    "Manifest missing required fields: version, type, check, and either streams or dynamic_streams"
  ],
  "warnings": [],
  "resolved_manifest": null
}

📚 Getting connector builder documentation...
📖 Documentation Overview:
  # Connector Builder Documentation

**Important**: Before starting development, call the `get_connector_builder_checklist()` tool first to get the comprehensive development checklist.

The checklist pr...

🤖 Demo 2: LLM Integration
==================================================
2025-08-12 15:34:07,207 - mcp_use.telemetry.telemetry - INFO - Anonymized telemetry enabled. Set MCP_USE_ANONYMIZED_TELEMETRY=false to disable.
💭 Asking LLM to validate and analyze the manifest...
2025-08-12 15:34:07,208 - mcp_use - INFO - 🚀 Initializing MCP agent and connecting to services...
2025-08-12 15:34:07,208 - mcp_use - INFO - 🔌 Found 0 existing sessions
2025-08-12 15:34:07,208 - mcp_use - INFO - 🔄 No active sessions found, creating new ones...
<frozen abc>:106: ExperimentalClassWarning: This class is experimental. Use at your own risk.
ExperimentalClassWarning: This class is experimental. Use at your own risk.
Starting Builder MCP server.
[08/12/25 15:34:08] INFO     Starting MCP server 'connector-builder-mcp' with transport 'stdio'            server.py:1307
2025-08-12 15:34:08,531 - mcp_use - INFO - ✅ Created 1 new sessions
2025-08-12 15:34:08,591 - mcp_use - INFO - 🛠️ Created 11 LangChain tools from client
2025-08-12 15:34:08,591 - mcp_use - INFO - 🧰 Found 11 tools across all connectors
2025-08-12 15:34:08,591 - mcp_use - INFO - 🧠 Agent ready with tools: validate_manifest, execute_stream_test_read, execute_record_counts_smoke_test, execute_dynamic_manifest_resolution_test, get_manifest_yaml_json_schema, get_connector_builder_checklist, get_connector_builder_docs, get_connector_manifest, find_connectors_by_class_name, list_dotenv_secrets, populate_dotenv_missing_secrets_stubs
2025-08-12 15:34:08,609 - mcp_use - INFO - ✨ Agent initialization complete
2025-08-12 15:34:08,609 - mcp_use - INFO - 💬 Received query: 'Please validate this connector manifest and provid...'
2025-08-12 15:34:08,609 - mcp_use - INFO - 🏁 Starting agent execution with max_steps=5
2025-08-12 15:34:08,609 - mcp_use - INFO - 👣 Step 1/5
2025-08-12 15:34:11,970 - mcp_use - INFO - 🔧 Tool call: validate_manifest with input: {'manifest': 'version: 4.6.2\ntype: DeclarativeSource\ncheck:\n  type: CheckStream\n  stream_name...
2025-08-12 15:34:11,970 - mcp_use - INFO - 📄 Tool result: {   "is_valid": false,   "errors": [     "Manifest missing required fields: version, type, check,...
2025-08-12 15:34:11,970 - mcp_use - INFO - 👣 Step 2/5
2025-08-12 15:34:15,441 - mcp_use - INFO - ✅ Agent finished at step 2
2025-08-12 15:34:15,441 - mcp_use - INFO - 🎉 Agent execution complete in 8.233030796051025 seconds
🤖 LLM Analysis:
  The validation of the connector manifest has been completed. Here are the results:

**Validation Result:**
- **Is Valid:** No
- **Errors:**
  - The manifest is missing required fields: version, type, check, and either streams or dynamic_streams.

**Feedback on Structure:**
1. **Required Fields:** Ensure that the manifest includes all required fields. In this case, it seems that the manifest is missing some essential components.
2. **Stream Definition:** The `streams` section is present, but the validation indicates that it may not be recognized correctly due to the missing required fields.
3. **Connection Specification:** The `connection_specification` is defined, but it is empty. You may want to include properties that are relevant to your connector.

To resolve the validation issues, please ensure that all required fields are included and correctly structured. You may want to refer to the official documentation for the specific requirements of a connector manifest.

⚙️  Demo 3: Multi-Tool Workflow
==================================================
<frozen abc>:106: ExperimentalClassWarning: This class is experimental. Use at your own risk.
ExperimentalClassWarning: This class is experimental. Use at your own risk.
Starting Builder MCP server.
[08/12/25 15:34:16] INFO     Starting MCP server 'connector-builder-mcp' with transport 'stdio'            server.py:1307
1️⃣  Validating manifest...
   ✅ Manifest validation complete

2️⃣  Getting development checklist...
   📋 Development checklist retrieved

3️⃣  Getting manifest JSON schema...
   📄 JSON schema retrieved

🎉 Multi-tool workflow completed successfully!
   This demonstrates how mcp-use can orchestrate multiple
   connector-builder-mcp tools in a single workflow.

============================================================
✨ Demo completed!
Builder MCP server stopped.
Builder MCP server stopped.
Builder MCP server stopped.
AJ-Steers-MacBook-Pro---GGHWM7QWPJ:builder-mcp ajsteers$ 

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[github-actions]

PyTest Results (Full)

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit 0df9fef. ± Comparison against base commit adf33bf.

♻️ This comment has been updated with latest results.