Skip to content

Refactor the evaluation results handling and upload to Braintrust#67

Merged
haoranpb merged 22 commits into
mainfrom
features/result-collection-upload
Nov 7, 2025
Merged

Refactor the evaluation results handling and upload to Braintrust#67
haoranpb merged 22 commits into
mainfrom
features/result-collection-upload

Conversation

@haoranpb

@haoranpb haoranpb commented Nov 6, 2025

Copy link
Copy Markdown
Collaborator

Refactoring the evaluation result into a different module with some improvements, address some other concerns along the way:

  • Better exception handling with Timeout

The eval results should be uploaded to:

  • Braintrust with bceval

Results are summarized into evaluation_summary.json

@haoranpb haoranpb requested a review from Copilot November 6, 2025 18:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors the evaluation results handling by extracting the result-related functionality from bcbench.evaluate into a new bcbench.results module, and introduces a new CLI command group for result operations. The changes improve code organization and add support for exporting results to bceval format with Braintrust integration.

Key changes:

  • Moved EvaluationResult and EvaluationResultSummary from evaluate/evaluation_result.py to results/evaluation_result.py
  • Split display logic into dedicated results/display.py module
  • Added results/result_writer.py for bceval format export
  • Moved evaluate summarize command to new result summarize command in commands/result.py
  • Updated config to dynamically load instance pattern from dataset schema for result file filtering
  • Added Braintrust upload step to GitHub workflow with bceval integration

Reviewed Changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/bcbench/results/evaluation_result.py New location for EvaluationResult and EvaluationResultSummary classes with from_json and from_results factory methods
src/bcbench/results/display.py Extracted display logic (console and GitHub summaries) into dedicated module
src/bcbench/results/result_writer.py New module for writing results in bceval format
src/bcbench/results/init.py Public API exports for results module
src/bcbench/commands/result.py New CLI command group with result summarize command
src/bcbench/config.py Added dynamic instance_pattern loading from dataset schema and result_pattern to FilePatternConfig
tests/test_evaluation_summary.py Updated tests to use new module paths and EvaluationResult objects instead of raw dicts
src/bcbench/evaluate/evaluation_pipeline.py Updated imports and result file naming to use config pattern
docs/_data/results.json Added leaderboard results data
evaluator/scores.py New bceval evaluator for resolution rate metric
.github/workflows/*.yml Updated workflows to use new result summarize command and added Braintrust integration

Comment thread src/bcbench/config.py
Comment thread .github/workflows/test-summarize-results.yml Fixed
Comment thread .github/workflows/test-summarize-results.yml Fixed
@haoranpb haoranpb changed the title Upload evaluation summary result Refactor the evaluation results handling and upload to Braintrust Nov 7, 2025
@haoranpb haoranpb marked this pull request as ready for review November 7, 2025 10:19
@haoranpb haoranpb merged commit 476c77e into main Nov 7, 2025
4 checks passed
@haoranpb haoranpb deleted the features/result-collection-upload branch November 7, 2025 10:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants