Skip to content

Introduce plugins support in BC-Bench#710

Draft
haoranpb with Copilot wants to merge 3 commits into
mainfrom
copilot/expand-experimentconfig-for-plugins
Draft

Introduce plugins support in BC-Bench#710
haoranpb with Copilot wants to merge 3 commits into
mainfrom
copilot/expand-experimentconfig-for-plugins

Conversation

Copilot AI commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

This PR is a prototype by Copilot, not verified.

The goal: Extends experiment configuration to install Copilot/Claude plugins, it should work across both Copilot CLI and Claude Code, with marketplaces pinned to a specific commit for reproducibility.

Config schema

TBD, prototype by coding agent.

plugins.install is a list of entries, each pinning a marketplace to a SHA:

plugins:
  install: []
    # - repo: "github/awesome-copilot"   # marketplace OWNER/REPO or git URL
    #   sha: "28c3a14af4e6232091071ddb40272f72d9d96b2f"
    #   name: "awesome-copilot@awesome-copilot"  # <plugin>@<marketplace>

Goal

BC-Bench must support experimentation with plugins (hosted in a public GitHub repo). In the https://github.com/microsoft/BC-Bench/blob/main/src/bcbench/agent/shared/config.yaml, there should be an entry for plugins.

By default, no plugins are enabled. And there should be a sample one (commented out) for folks to easily test out.

The orchestration should be handled by coding agents (e.g. Copilot CLI) as much as possible => less code here in BC-Bench.

Testing

Before merging, this must be tested (using a public plugin from GitHub maybe):

  1. Locally using the sample
  2. Tested with few entries, using both GitHub Copilot and Claude Code
  3. A full test run in Code Review category, and observe the Kusto storage is correct uploaded

Co-authored-by: haoranpb <27280733+haoranpb@users.noreply.github.com>
…ilot/expand-experimentconfig-for-plugins
Co-authored-by: haoranpb <27280733+haoranpb@users.noreply.github.com>
Copilot AI changed the title Support Copilot plugins in ExperimentConfiguration Support cross-CLI, SHA-pinned plugins in ExperimentConfig Jun 30, 2026
Copilot AI requested a review from haoranpb June 30, 2026 12:35
@haoranpb haoranpb changed the title Support cross-CLI, SHA-pinned plugins in ExperimentConfig Introduce plugins support in BC-Bench Jul 2, 2026
@haoranpb haoranpb assigned Groenbech96 and unassigned haoranpb and Copilot Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants