feat: implement dotenv-based secret management system

[aaronsteers]

feat: implement dotenv-based secret management system

Summary

This PR implements a comprehensive dotenv-based secret management system for connector-builder-mcp that allows users to manage secrets in .env files without exposing actual secret values to the LLM. The system provides MCP tools for managing secrets and automatically hydrates connector configurations with secrets during execution.

Key Features:

• File-based secret management: Uses .env files with configurable paths via environment variable or MCP tool
• Naming convention mapping: Environment variables map to config paths (e.g., CREDENTIALS_PASSWORD → credentials.password)
• MCP tools: set_dotenv_path, list_secrets, add_secret_stub, get_secrets_file_path_for_user
• Automatic config hydration: Seamlessly injects secrets into connector configs before execution
• Security-first: LLM never sees actual secret values, only metadata about what secrets exist

Implementation Details:

• Added python-dotenv>=1.0.0 dependency
• Created _secrets.py module with global state management for dotenv file path
• Integrated config hydration into all three connector builder operations: validate_manifest, execute_stream_test_read, and get_resolved_manifest
• Comprehensive test suite with 18 test cases covering all functionality

Review & Testing Checklist for Human

• End-to-end workflow test: Use MCP client to set dotenv path, add secret stubs, populate with real values, and verify connector operations work correctly
• Naming convention verification: Test various environment variable patterns (API_KEY, CREDENTIALS_PASSWORD, OAUTH_CLIENT_SECRET) map correctly to config paths
• Config hydration integration: Verify existing connector operations still work and secrets are properly injected without breaking functionality
• Tool registration: Confirm all four new MCP tools are accessible and work as expected
• File handling edge cases: Test with various file paths, missing directories, permissions issues, and concurrent access scenarios

Recommended Test Plan:

1. Start MCP server with poe server and connect with MCP client
2. Use set_dotenv_path to configure a test .env file
3. Use add_secret_stub to add placeholder secrets
4. Manually edit the .env file to add real secret values
5. Run connector validation/testing operations and verify secrets are hydrated correctly
6. Test list_secrets shows secret keys without exposing values

---

Diagram

%%{ init : { "theme" : "default" }}%%
graph TD
    Client["MCP Client"]
    Server["connector_builder_mcp/server.py"]:::context
    ConnectorBuilder["_connector_builder.py"]:::major-edit
    Secrets["_secrets.py"]:::major-edit
    EnvFile[".env file"]
    Tests["tests/test_secrets.py"]:::major-edit
    PyProject["pyproject.toml"]:::minor-edit
    Util["_util.py"]:::minor-edit
    
    Client --> Server
    Server --> ConnectorBuilder
    ConnectorBuilder --> Secrets
    Secrets --> EnvFile
    Tests --> Secrets
    
    ConnectorBuilder -.->|"hydrate_config()"| Secrets
    Secrets -.->|"load_secrets()"| EnvFile
    Client -.->|"set_dotenv_path<br/>list_secrets<br/>add_secret_stub"| Secrets

    subgraph Legend
        L1[Major Edit]:::major-edit
        L2[Minor Edit]:::minor-edit
        L3[Context/No Edit]:::context
    end

    classDef major-edit fill:#90EE90
    classDef minor-edit fill:#87CEEB
    classDef context fill:#FFFFFF

Loading

Notes

• The _env_var_to_path function contains complex logic for mapping environment variable names to config paths - this is a critical piece that should be thoroughly tested
• Global state management with _current_dotenv_path could potentially have concurrency issues in multi-threaded scenarios
• Config hydration happens automatically before every connector operation, so any bugs here could break existing functionality
• The implementation assumes users will follow the naming convention correctly - consider adding validation or better error messages for misnamed variables

Link to Devin run: https://app.devin.ai/sessions/eab82b57cce84fb8aeb33b11e2aabea0
Requested by: AJ Steers (@aaronsteers)

[devin-ai-integration]

Original prompt from AJ Steers

@Devin - Add a dotenv implementation to connector-builder-mcp. Specifically, here are the details of how I want you to implement it. Instead of passing the environment variables to the server directly, we're going to send a file pointer as an environment variable. The reason for this is that we're not going to give the LLM access to the secrets, we are only going to have tools that list secrets in the file, add stubs for new secrets that are needed from the user, print the file path to the user, etc. 

Our tool will have access to those secrets via the Python dotenv library, and we'll hydrate connector config as needed, given the secrets that are provided. In this way the LM can build the connector, without ever getting access to the secrets. And the user is told exactly what they need to provide, and we make it easy for them to provide those secrets within the file that we give them the path to.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Testing This Branch via MCP

To test the changes in this specific branch with an MCP client like Claude Desktop, use the following configuration:

{
  "mcpServers": {
    "connector-builder-mcp-dev": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/airbytehq/connector-builder-mcp.git@devin/1754097884-dotenv-implementation", "connector-builder-mcp"]
    }
  }
}

Testing This Branch via CLI

You can test this version of the MCP Server using the following CLI snippet:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/connector-builder-mcp.git@devin/1754097884-dotenv-implementation#egg=airbyte-connector-builder-mcp' --help

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poe <command> - Runs any poe command in the uv virtual environment

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit db0964e. ± Comparison against base commit 65b10d8.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit db0964e. ± Comparison against base commit 65b10d8.

♻️ This comment has been updated with latest results.

[Copilot]

Pull Request Overview

This PR implements a comprehensive dotenv-based secret management system that allows users to manage secrets in .env files without exposing actual secret values to the LLM. The system provides MCP tools for managing secrets and automatically hydrates connector configurations with secrets during execution.

• Adds file-based secret management using .env files with configurable paths
• Implements naming convention mapping from environment variables to config paths (e.g., CREDENTIALS_PASSWORD → credentials.password)
• Provides four MCP tools for secret management: set_dotenv_path, list_secrets, add_secret_stub, and get_secrets_file_path_for_user
• Integrates automatic config hydration into all three connector builder operations

Reviewed Changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
tests/test_secrets.py	Comprehensive test suite with 18 test cases covering all secrets functionality
pyproject.toml	Adds python-dotenv>=1.0.0 dependency
connector_builder_mcp/_util.py	Adds documentation note referencing the new secrets hydration functionality
connector_builder_mcp/_secrets.py	Core secrets management module with dotenv file handling, config hydration, and MCP tools
connector_builder_mcp/_connector_builder.py	Integrates config hydration into connector operations and registers secrets tools

[devin-ai-integration]

Test Implementation Audit - tests/test_secrets.py

I've completed the requested audit of the test implementation. Here are my findings:

Issues Identified

1. Test Classes vs Functions ✅ Action Required

All 5 test classes (TestLoadSecrets, TestHydrateConfig, TestListDotenvSecrets, TestPopulateDotenvMissingSecretsStubs, TestGetDotenvPath) are just organizational containers with no shared setup/teardown. As requested, these should be converted to standalone test functions for better readability.

2. Potentially Useless Tests ⚠️ 3 tests identified

• test_hydrate_config_empty_config (lines 99-102) - Tests trivial case hydrate_config({}, "/path") == {}
• test_hydrate_config_none_config (lines 104-107) - Tests trivial case hydrate_config(None, "/path") == None
• test_get_dotenv_path (lines 277-281) - Tests trivial wrapper around Path().absolute()

These tests don't add meaningful validation beyond what the function signature already guarantees.

3. Wasteful Redundancies ⚠️ 1 test identified

• test_hydrate_config_simple_keys (lines 77-86) overlaps significantly with test_hydrate_config_with_secrets (lines 56-75) - both test basic key injection functionality with similar patterns.

4. Suppressed Signals ✅ None found

All tests properly validate expected behavior and error conditions. No tests are inadvertently suppressing important error signals.

Summary

• Total tests: 18
• Tests to convert: All (classes → functions)
• Tests to remove: 3 useless tests
• Tests to consolidate: 1 redundant test
• Final test count: ~14 focused tests

Proposed Changes

1. Convert all test classes to standalone functions
2. Remove the 3 trivial tests that don't add value
3. Consolidate the redundant hydration test
4. Maintain all meaningful test coverage

The remaining tests provide good coverage of:

• Edge cases (missing files, empty manifests, etc.)
• Core functionality (secret loading, config hydration, stub generation)
• Error conditions (invalid inputs, file creation failures)

Would you like me to proceed with implementing these improvements?

[devin-ai-integration]

Test Implementation Improvements Complete ✅

I've successfully implemented the audit recommendations:

Changes Made

1. ✅ Converted all test classes to standalone functions - All 5 test classes (TestLoadSecrets, TestHydrateConfig, etc.) are now standalone functions for better readability
2. ✅ Removed 3 useless tests that didn't add meaningful value:
   • test_hydrate_config_empty_config - trivial case testing hydrate_config({}, "/path") == {}
   • test_hydrate_config_none_config - trivial case testing hydrate_config(None, "/path") == None
   • Kept test_get_dotenv_path as requested (though it's a simple wrapper test)
3. ✅ Consolidated redundant test - Merged test_hydrate_config_simple_keys into test_hydrate_config_with_secrets to eliminate duplication

Results

• Test count: Reduced from 18 to 15 focused tests
• All tests pass: ✅ 15/15 tests passing
• Code quality: ✅ All formatting, linting, and type checks pass
• Coverage maintained: All meaningful functionality still tested

Test Structure Now

tests/test_secrets.py (15 functions)
├── test_load_secrets_* (2 tests)
├── test_hydrate_config_* (5 tests) 
├── test_list_dotenv_secrets_* (2 tests)
├── test_populate_dotenv_missing_secrets_stubs_* (5 tests)
└── test_get_dotenv_path (1 test)

The tests are now more readable, focused, and eliminate the wasteful redundancies identified in the audit. No suppressed signals were found - all tests properly validate expected behavior and error conditions.