feat: Add privatebin URL support to connector-builder-mcp

[aaronsteers]

feat: Add pastebin URL support to connector-builder-mcp

Summary

This PR extends the connector-builder-mcp to support pastebin URLs as sources for environment variables, alongside the existing local .env file support. The implementation includes:

• Breaking Change: Renamed dotenv_path parameter to secrets_env_file_uris across all MCP tools (execute_stream_test_read, execute_record_counts_smoke_test, list_dotenv_secrets, populate_dotenv_missing_secrets_stubs)
• Pastebin URL Support: Added support for pastebin:// URLs (e.g., pastebin://pastebin.com/abc123) with password authentication via PASTEBIN_PASSWORD environment variable
• Array Support: Now accepts arrays of files/URLs ["file1.env", "pastebin://abc123"] and comma-separated strings as fallback
• Enhanced Error Handling: Modified populate_dotenv_missing_secrets_stubs to handle immutable pastebin URLs by providing manual update instructions instead of attempting to write

Review & Testing Checklist for Human

• Test pastebin functionality end-to-end with a real password-protected pastebin URL to verify the authentication and content fetching works correctly
• Verify security: Confirm that PASTEBIN_PASSWORD is not leaked in logs, error messages, or debug output when authentication fails
• Test error scenarios: Try invalid pastebin URLs, missing passwords, network failures, and verify graceful error handling
• Test mixed sources: Use both local .env files and pastebin URLs together in the same request to ensure proper merging of secrets
• Validate breaking change: Confirm that the parameter rename from dotenv_path to secrets_env_file_uris is acceptable and any downstream consumers can be updated

Recommended Test Plan:

1. Create a password-protected pastebin with test secrets
2. Set PASTEBIN_PASSWORD environment variable
3. Test execute_stream_test_read with the pastebin URL
4. Test list_dotenv_secrets with mixed local file + pastebin URL
5. Test populate_dotenv_missing_secrets_stubs with pastebin URL to verify instruction-based response

---

Diagram

%%{ init : { "theme" : "default" }}%%
graph TD
    CB["connector_builder_mcp/_connector_builder.py"]:::major-edit
    SEC["connector_builder_mcp/_secrets.py"]:::major-edit
    TEST["tests/test_secrets.py"]:::major-edit
    
    CB --> |"uses"| SEC
    SEC --> |"load_secrets()"| PARSE["_parse_secrets_uris()"]:::major-edit
    SEC --> |"pastebin URLs"| FETCH["_fetch_pastebin_content()"]:::major-edit
    FETCH --> |"HTTP GET with password"| PASTEBIN["pastebin.com"]:::context
    
    SEC --> |"hydrate_config()"| CONFIG["Connector Config"]:::context
    CB --> |"execute_stream_test_read<br/>execute_record_counts_smoke_test"| CONFIG
    
    subgraph Legend
        L1["Major Edit"]:::major-edit
        L2["Minor Edit"]:::minor-edit  
        L3["Context/No Edit"]:::context
    end

    classDef major-edit fill:#90EE90
    classDef minor-edit fill:#87CEEB
    classDef context fill:#FFFFFF

Loading

Notes

• All existing functionality for local .env files is preserved
• The dotenv_values() function required StringIO wrapper for pastebin content parsing
• Comprehensive test coverage added including mocked HTTP requests and various edge cases
• Error handling follows existing patterns (log errors, return empty dict, continue processing)

Link to Devin run: https://app.devin.ai/sessions/df7d37240e154ce5a3f708a3b9ba8233
Requested by: AJ Steers (@aaronsteers)

Important

Auto-merge enabled.

This PR is set to merge automatically when all requirements are met.

[devin-ai-integration]

Original prompt from AJ Steers

@Devin - let's add pastebin support to connector builder MCP. Everywhere that we currently support sending a dot ENV file to a tool, will update to accept an array of ENV files and or URLs. Let's decide before we start if we want to use a custom URI or if there is one already in existence URI prefix for paste bin versus local file. One important requirement is that we only support pastebin URLs that have a passcode or password set. The passphrase must be sent via environment variables to the MCP server, and this is the only way we can ensure that the LLM is not able to access the raw secret

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Testing This Branch via MCP

To test the changes in this specific branch with an MCP client like Claude Desktop, use the following configuration:

{
  "mcpServers": {
    "connector-builder-mcp-dev": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/airbytehq/connector-builder-mcp.git@devin/1755295126-pastebin-support", "connector-builder-mcp"]
    }
  }
}

Testing This Branch via CLI

You can test this version of the MCP Server using the following CLI snippet:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/connector-builder-mcp.git@devin/1755295126-pastebin-support#egg=airbyte-connector-builder-mcp' --help

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poe <command> - Runs any poe command in the uv virtual environment

📝 Edit this welcome message.

[Copilot]

Pull Request Overview

This PR adds support for pastebin URLs as sources for environment variables alongside existing local .env file support in the connector-builder-mcp. It introduces a breaking change by renaming the dotenv_path parameter to secrets_env_file_uris to better reflect the expanded functionality.

• Renamed dotenv_path parameter to secrets_env_file_uris across all MCP tools
• Added pastebin URL support with pastebin:// scheme and password authentication via PASTEBIN_PASSWORD
• Enhanced input handling to accept arrays of files/URLs and comma-separated strings

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
connector_builder_mcp/_secrets.py	Major refactoring to support pastebin URLs, new parsing functions, and enhanced error handling
connector_builder_mcp/_connector_builder.py	Updated parameter names and descriptions for the breaking change
tests/test_secrets.py	Comprehensive test coverage for new pastebin functionality and edge cases

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[github-actions]

PyTest Results (Fast)

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit 8f8f1e3. ± Comparison against base commit c5b6665.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit 8f8f1e3. ± Comparison against base commit c5b6665.

♻️ This comment has been updated with latest results.