feat: add evals using arize phoenix

[pedroslopez]

Evaluation framework and experiment automation:

• Added a new phoenix_run.py module to automate connector builder evaluations using the Phoenix framework, including experiment orchestration, dataset management, and evaluator integration.
• Added dataset.py to load and manage evaluation datasets from a YAML config, including Phoenix dataset creation and hashing for versioning.
• Introduced evaluators.py with LLM-based readiness and stream coverage evaluators
• Added task.py to define the connector build task for experiments, including artifact collection and result formatting.
• Added helpers.py for reading artifacts from the workspace directory.
• Created a YAML dataset listing connectors and expected streams for evaluation in data/connectors.yaml.

Agent build pipeline refactor:

• Refactored run_connector_build and run_manager_developer_build in run.py to return lists of RunResult objects instead of None, enabling collection and evaluation of build results. Now handles errors gracefully and returns partial results if interrupted. [1] [2] [3] [4] [5] [6] [7]

Dependency and CLI updates:

• Added required dependencies for Phoenix evaluation, pandas, YAML, and OpenInference instrumentation in pyproject.toml.
• Added a new CLI task run-evals in poe_tasks.toml to run the evaluation workflow.

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Testing This Branch via MCP

To test the changes in this specific branch with an MCP client like Claude Desktop, use the following configuration:

{
  "mcpServers": {
    "connector-builder-mcp-dev": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/airbytehq/connector-builder-mcp.git@pedro/evals", "connector-builder-mcp"]
    }
  }
}

Testing This Branch via CLI

You can test this version of the MCP Server using the following CLI snippet:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/connector-builder-mcp.git@pedro/evals#egg=airbyte-connector-builder-mcp' --help

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poe <command> - Runs any poe command in the uv virtual environment
• /poe build-connector prompt="Star Wars API" - Run the connector builder using the Star Wars API.

📝 Edit this welcome message.

[github-actions]

PyTest Results (Full)

0 tests   0 ✅  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit 8866540.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Fast)

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit 8866540. ± Comparison against base commit 6624d45.

♻️ This comment has been updated with latest results.

[aaronsteers]

Everything here looks good to me - especially for a first iteration, I think this is strong. 💪

Approving for merge when ready - or re-request my review if you make substantive changes that need another pair of eyes. Thanks!

[aaronsteers]

Not a blocker for the PR, but we can use this space for eval module-level docs. Specifically, the docstring we add here will get rendered in the autogenerated pdoc docs.