Skip to content

Conversation

aaronsteers
Copy link
Contributor

Upgrade poe-command-processor to v1.3.3 for log-capture support

Summary

Upgrades the poe-command-processor action from v1 to v1.3.3 across three workflow files to enable automatic log-capture functionality. The v1.3.3 release (published today) adds auto-inclusion of GITHUB_STEP_SUMMARY content in slash command replies.

Key benefit: The evals code already writes markdown summaries to GITHUB_STEP_SUMMARY. With this upgrade, eval summaries will automatically appear in PR comments as collapsible details sections, instead of only being visible in workflow logs.

Affected workflows:

  • .github/workflows/poe-command.yml - Generic poe command dispatcher
  • .github/workflows/build-connector-command.yml - Connector build/eval workflow
  • .github/workflows/run-standard-evals.yml - Scheduled eval runs

Review & Testing Checklist for Human

  • Test the new feature: Create a test PR and run /build-connector source-jsonplaceholder to verify that eval summaries appear in the PR comment with score distributions, per-connector tables, and delta comparisons
  • Verify no breaking changes: Review the release notes from v1.1.x through v1.3.3 to confirm backward compatibility
  • Monitor scheduled evals: Check that the next scheduled eval run (Mon/Wed/Fri at 00:00 UTC) completes successfully and includes the summary in any created issues/comments
  • Check formatting: Verify that the collapsible <details> summary section renders correctly in PR comments and isn't excessively verbose

Test Plan

  1. Create a test PR in this repo
  2. Comment /build-connector source-jsonplaceholder on the PR
  3. Wait for the workflow to complete
  4. Verify a collapsible "Show/Hide Summary Output" section appears in the bot's reply
  5. Expand the section and confirm it shows eval results with emoji indicators, score tables, etc.

Notes

  • The v1.3.3 release was published today (Oct 12, 2025), so this is a brand-new feature
  • The evals code already has the necessary GITHUB_STEP_SUMMARY instrumentation (see connector_builder_agents/src/evals/summary.py:726-734)
  • No code changes were needed - this is purely a version bump to enable the new feature

Link to Devin run: https://app.devin.ai/sessions/e0d5fac6911c4ed18a4dad393546a14c
Requested by: @aaronsteers

Copy link
Contributor

Original prompt from AJ Steers
@Devin - evaluate our slash command usage and especially any actions using the poe-command-processor action. Then suggest actions that could benefit from the new log-capture feature added in the latest release.
Thread URL: https://airbytehq-team.slack.com/archives/D089P0UPVT4/p1760244149411999?thread_ts=1760244149.411999

Copy link
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions github-actions bot added the chore label Oct 12, 2025
Copy link

👋 Welcome to the Airbyte Connector Builder MCP!

Thank you for your contribution! Here are some helpful tips and reminders for your convenience.

Testing This Branch via MCP

To test the changes in this specific branch with an MCP client like Claude Desktop, use the following configuration:

{
  "mcpServers": {
    "connector-builder-mcp-dev": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/airbytehq/connector-builder-mcp.git@devin/1760245165-upgrade-poe-command-processor-v1.3.3", "connector-builder-mcp"]
    }
  }
}

Testing This Branch via CLI

You can test this version of the MCP Server using the following CLI snippet:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/connector-builder-mcp.git@devin/1760245165-upgrade-poe-command-processor-v1.3.3#egg=airbyte-connector-builder-mcp' --help

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /build-connector - Builds the default connector on-demand using the AI builder
  • /build-connector prompt="<your prompt>" - Builds a connector on-demand using the AI builder
  • /poe <command> - Runs any poe command in the uv virtual environment

AI Builder Evaluations

AI builder evaluations run automatically under the following conditions:

  • When a PR is marked as "ready for review"
  • When a PR is reopened

A set of standardized evaluations also run on a schedule (Mon/Wed/Fri at midnight UTC) and can be manually triggered via workflow dispatch.

Helpful Resources

If you have any questions, feel free to ask in the PR comments or join our Slack community.

📝 Edit this welcome message.

@aaronsteers aaronsteers merged commit 45b1b8c into main Oct 12, 2025
13 checks passed
@aaronsteers aaronsteers deleted the devin/1760245165-upgrade-poe-command-processor-v1.3.3 branch October 12, 2025 05:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant