Skip to content

docs: clean up CONTRIBUTING.md and other docs #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Sep 15, 2025
Merged

docs: clean up CONTRIBUTING.md and other docs #74

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d3f2e7e
docs: clean-up CONTRIBUTING.md
aaronsteers Sep 15, 2025
51c3c9a
clean up readme
aaronsteers Sep 15, 2025
1559ad0
consolidate TESTING.md into CONTRIBUTING.md
aaronsteers Sep 15, 2025
348ed29
clean up wording
aaronsteers Sep 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
240 changes: 108 additions & 132 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,37 +9,22 @@ This project uses [uv](https://docs.astral.sh/uv/) for Python package management
### Prerequisites

- Python 3.10+
- [uv](https://docs.astral.sh/uv/) for package management
- [uv](https://docs.astral.sh/uv/) for package management (`brew install uv`)

### Installing Dependencies

```bash
# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --all-extras
# Or:
poe sync
# Verify installation:
uv run connector-builder-mcp --help
```

### Quick Start

1. **Clone the repository**:

```bash
git clone https://github.com/airbytehq/connector-builder-mcp.git
cd connector-builder-mcp
```

2. **Install dependencies**:
_(Note: Unlike Poetry, uv will generally auto-run a sync whenever you use `uv run`. So, `uv sync`
may not be strictly necessary.)_

```bash
uv sync --all-extras
# Or:
poe sync
```

3. **Verify the installation**:

```bash
uv run connector-builder-mcp --help
```

### Using Poe Tasks
### Installing Poe

For convenience, install [Poe the Poet](https://poethepoet.natn.io/) task runner:

Expand All @@ -51,109 +36,144 @@ uv tool install poethepoet
poe --help
```

Available Poe commands:

```bash
# Development
poe install # Install dependencies
poe sync # Alias for install

# Code quality
poe format # Format code with ruff
poe lint # Lint code with ruff
poe lint-fix # Lint and auto-fix issues
poe typecheck # Type check with mypy
poe check # Run all checks (lint + typecheck + test)
## Helpful Poe Shortcuts

# Testing
poe test # Run tests with verbose output
poe test-fast # Run tests, stop on first failure
Note: The below is _not_ a full list of poe commands. For a full list, run `poe --help`.

```bash
# MCP server operations
poe mcp-serve-local # Serve with STDIO transport
poe mcp-serve-http # Serve over HTTP
poe mcp-serve-sse # Serve over SSE
poe inspect # Inspect available MCP tools

# Combined workflows
poe check # Run lint + typecheck + test
poe inspect # Inspect the MCP server. Use --help for options.
poe inspect --tools # Inspect the tools.
poe test-tool # Spin up server, pass a tool call, then spin down the server.
poe agent-run # Run a connector build using the Pytest test script.
```

You can see what any Poe task does by checking the `poe_tasks.toml` file at the root of the repo.

## Manual Commands (without Poe)
## Testing with GitHub Models

```bash
brew install gh
gh auth login
gh extension install https://github.com/github/gh-models
gh models --help
```

If you prefer to run commands directly with uv:
## Running Pytest Tests

```bash
# Development
uv sync --all-extras # Install dependencies
uv run ruff format . # Format code
uv run ruff check . # Lint code
uv run mypy connector_builder_mcp # Type checking
uv run pytest tests/ -v # Run tests

# MCP server
uv run connector-builder-mcp # Start server
poe inspect # Inspect tools
# Make sure dependencies are up-to-date
poe sync

# Run all tests
poe test

# Run only integration tests
uv run pytest tests/test_integration.py -v

# Run tests requiring credentials (skipped by default)
uv run pytest tests/ -v -m requires_creds

# Run fast tests only (skip slow integration tests)
uv run pytest tests/ -v -m "not requires_creds"
```

## Testing
## MCP Server Inspection

Inspect the MCP server to see available tools, resources, and prompts:

```bash
# Inspect the server structure (generates comprehensive JSON report)
poe inspect
# Equivalent to: uv run fastmcp inspect connector_builder_mcp/server.py:app

# Save inspection report to custom file
poe inspect --output my-server-report.json
# Equivalent to: uv run fastmcp inspect connector_builder_mcp/server.py:app --output my-server-report.json

# View help for inspection options
poe inspect --help
# Shows available options for the inspect command
```

The project includes comprehensive tests covering:
The inspection generates a comprehensive JSON report containing: **Tools**, **Prompts**, **Resources**, **Templates**, and **Capabilities**.

- **Server functionality**: MCP server initialization and tool registration
- **Connector builder tools**: Manifest validation, stream testing, and resolution
- **Utility functions**: Configuration filtering and validation
### Inspecting Specific Tools

Run the full test suite:
After running `poe inspect`, you can examine the generated `server-info.json` file to see detailed information about each tool:

```bash
poe test
# or
uv run pytest tests/ -v
# View the complete inspection report
cat server-info.json

# Extract just the tools information using jq
cat server-info.json | jq '.tools'

# Get details for a specific tool
cat server-info.json | jq '.tools[] | select(.name == "validate_manifest")'
```

### Testing with Real Connectors
## Testing MCP Tools

Test individual MCP tools directly with JSON arguments using the `test-tool` command:

```bash
# Test manifest validation
poe test-tool validate_manifest '{"manifest": {"version": "4.6.2", "type": "DeclarativeSource"}, "config": {}}'

# Test secrets listing with local file
poe test-tool list_dotenv_secrets '{"dotenv_path": "/absolute/path/to/.env"}'

To test with actual Airbyte connector manifests:
# Test populating missing secrets
poe test-tool populate_dotenv_missing_secrets_stubs '{"dotenv_path": "/path/to/.env", "config_paths": "api_key,secret_token"}'
```

1. Prepare a connector manifest (JSON format)
2. Use the MCP tools through the server or test them directly
3. Verify that validation, stream testing, and resolution work as expected
The `poe test-tool` command is ideal for:

## Code Style
- Quick testing of individual tools during development
- Testing with real data without setting up full MCP client
- Debugging tool behavior with specific inputs
- Validating privatebin URL functionality

We use [Ruff](https://docs.astral.sh/ruff/) for both linting and formatting:
## Using PrivateBin for Connector Config Secrets

- **Line length**: 100 characters
- **Target Python version**: 3.10+
- **Import sorting**: Automatic with ruff
- **Type hints**: Required for all public functions
PrivateBin can be used when it's not feasible to have a local `.env` file. When using PrivateBin:

Before submitting changes:
1. When creating the privatebin Secret, simply use the same format as you would for a `.env` file.
2. Always set a constant text password as an additional encryption layer. (Use the same password across all files you will use in a given session.)
3. Pass the password as an env var. (Don't give it to the agent.)
4. Private an expiration window such as 1 day or 1 week, depending on your requirements.

```bash
poe check # Runs formatting, linting, type checking, and tests
# Test secrets listing with privatebin URL (requires PRIVATEBIN_PASSWORD env var)
export PRIVATEBIN_PASSWORD="your_password"
poe test-tool list_dotenv_secrets '{"dotenv_path": "https://privatebin.net/?abc123"}'

# Test with privatebin URL
poe test-tool populate_dotenv_missing_secrets_stubs '{"dotenv_path": "https://privatebin.net/?abc123#passphrase", "config_paths": "api_key,secret_token"}'
```

## MCP Tool Development
## Testing with the VS Code MCP Extension

When adding new MCP tools:
The repository includes a pre-configured MCP setup in `.vscode/mcp.json`. Install the MCP extension and use the command palette to access connector builder tools directly in your editor.

1. **Add the tool function** in the appropriate module (e.g., `connector_builder_mcp/_connector_builder.py`)
2. **Use proper type annotations** with Pydantic models for complex inputs/outputs
3. **Register the tool** in the module's registration function
4. **Add comprehensive tests** covering success and failure cases
5. **Update documentation** if the tool adds new capabilities
## MCP Tools Dev Guide

This section has tools on how to develop MCP tools.

### Tool Function Pattern

Here is an example tool definition.

```python
from typing import Annotated
from pydantic import Field
from fastmcp import FastMCP

# @app.tool # deferred
def my_new_tool(
param: Annotated[
str,
Expand All @@ -176,51 +196,6 @@ def register_tools(app: FastMCP) -> None:
app.tool()(my_new_tool)
```

## AI Ownership Focus

This project emphasizes **AI ownership** rather than AI assistance. When contributing:

- **Design for autonomy**: Tools should enable end-to-end AI control of connector building
- **Comprehensive error handling**: AI agents need clear error messages and recovery paths
- **Structured outputs**: Use Pydantic models for consistent, parseable responses
- **Testing workflows**: Include tools that support automated testing and validation

## Submitting Changes

1. **Create a feature branch**: `git checkout -b feature/your-feature-name`
2. **Make your changes** following the code style guidelines
3. **Run the full test suite**: `poe check`
4. **Commit with clear messages**: Use conventional commit format
5. **Push and create a pull request**

### Commit Message Format

We follow [Conventional Commits](https://www.conventionalcommits.org/):

```text
feat: add new MCP tool for stream discovery
fix: resolve manifest validation edge case
docs: update contributing guidelines
test: add integration tests for connector builder
```

## Getting Help

- **Issues**: Report bugs and request features via GitHub Issues
- **Discussions**: Use GitHub Discussions for questions and ideas
- **Code Review**: All changes require review before merging

Thank you for contributing to Builder MCP! 🚀

## Testing with GitHub Models

```bash
brew install gh
gh auth login
gh extension install https://github.com/github/gh-models
gh models --help
```

## Debugging

One or more of these may be helpful in debugging:
Expand All @@ -230,4 +205,5 @@ export HTTPX_LOG_LEVEL=debug
export DEBUG='openai:*'
export OPENAI_AGENTS_LOG=debug
export OPENAI_LOG=debug
export FASTMCP_DEBUG=1
```
55 changes: 0 additions & 55 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,40 +13,6 @@ A Model Context Protocol (MCP) server for Airbyte connector building operations,
- **Configuration Management**: Validate connector configurations
- **Test Execution**: Run connector tests with proper limits and constraints

## Quick Start

**Prerequisites:**

- [uv](https://docs.astral.sh/uv/) for package management (`brew install uv`)
- Python 3.10+ (`uv python install 3.10`)

If you are developing or testing locally, you will also want to install:

- [PoeThePoet](https://poethepoet.natn.io) as a task manager (`uv tool install poethepoet`)

*See the [Contributing Guide](CONTRIBUTING.md) or [Testing Guide](TESTING.md) for more information about working with the repo locally.*

**Install:**

The Poe `sync` and `install` commands are identical, giving a quick way to update your virtual environment or create one from scratch, if needed.

```bash
# These are identical:
uv sync --all-extras
poe install
poe sync
```

**Run:**

```bash
# You can use any of these to start the server manually:
uv run connector-builder-mcp
poe mcp-serve-local
poe mcp-serve-http
poe mcp-serve-sse
```

## MCP Client Configuration

To use with MCP clients like Claude Desktop, add the following configuration:
Expand Down Expand Up @@ -108,24 +74,3 @@ For VS Code users with the MCP extension, use the included configuration in `.vs

- **[Contributing Guide](./CONTRIBUTING.md)** - Development setup, workflows, and contribution guidelines
- **[Testing Guide](./TESTING.md)** - Comprehensive testing instructions and best practices

### Using Poe Tasks

For convenience, you can use [Poe the Poet](https://poethepoet.natn.io/) task runner:

```bash
# Install Poe
uv tool install poethepoet

# Then use ergonomic commands
poe install # Install dependencies
poe check # Run all checks (lint + typecheck + test)
poe test # Run tests
poe mcp-serve-local # Serve locally
poe mcp-serve-http # Serve over HTTP
poe mcp-serve-sse # Serve over SSE
```

You can also run `poe --help` to see a full list of available Poe commands.

If you ever want to see what a Poe task is doing (such as to run it directly or customize how it runs), check out the `poe_tasks.toml` file at the root of the repo.
Loading
Loading