Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -266,3 +266,14 @@ dmypy.json
# Evaluation Result files
evaluation_results/
verification_results/

# Jekyll (GitHub Pages) - docs folder
docs/_site/
docs/.sass-cache/
docs/.jekyll-cache/
docs/.jekyll-metadata
docs/vendor/
docs/.bundle/
docs/tmp
docs/Gemfile
docs/Gemfile.lock
4 changes: 4 additions & 0 deletions docs/_config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
title: BC-Bench
description: Inspired by SWE-Bench, for Business Central (AL) ecosystem.
show_downloads: false
theme: jekyll-theme-cayman
52 changes: 52 additions & 0 deletions docs/_data/leaderboard.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
[
{
"agent_name": "GitHub Copilot CLI",
"model": "claude-sonnet-4",
"total": 19,
"resolved": 7,
"average_duration": 0,
"average_prompt_tokens": 0,
"average_completion_tokens": 0,
"date": "2025-11-05"
},
{
"agent_name": "GitHub Copilot CLI",
"model": "claude-sonnet-4.5",
"total": 19,
"resolved": 9,
"average_duration": 0,
"average_prompt_tokens": 0,
"average_completion_tokens": 0,
"date": "2025-11-04"
},
{
"agent_name": "GitHub Copilot CLI",
"model": "claude-haiku-4.5",
"total": 19,
"resolved": 10,
"average_duration": 0,
"average_prompt_tokens": 0,
"average_completion_tokens": 0,
"date": "2025-11-04"
},
{
"agent_name": "GitHub Copilot CLI",
"model": "gpt-5",
"total": 19,
"resolved": 5,
"average_duration": 0,
"average_prompt_tokens": 0,
"average_completion_tokens": 0,
"date": "2025-11-04"
},
{
"agent_name": "mini-bc-agent",
"model": "azure/gpt-4.1",
"total": 19,
"resolved": 6,
"average_duration": 0,
"average_prompt_tokens": 0,
"average_completion_tokens": 0,
"date": "2025-11-05"
}
]
38 changes: 38 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
layout: default
title: BC-Bench
---

## Evaluation Results

The table below shows the performance of different AI coding agents on BC-Bench tasks. Results are sorted by accuracy (number of resolved tasks).

<table>
<thead>
<tr>
<th>Agent (Model)</th>
<th>Accuracy</th>
<th>Avg Duration (s)</th>
<th>Token Consumption</th>
<th>Date</th>
</tr>
</thead>
<tbody>
{% assign sorted_results = site.data.leaderboard | sort: "resolved" | reverse %}
{% for result in sorted_results %}
<tr>
<td><strong>{{ result.agent_name }}</strong> ({{ result.model }})</td>
<td>{{ result.resolved }} / {{ result.total }} ({{ result.resolved | times: 100.0 | divided_by: result.total | round: 1 }}%)</td>
<td>{{ result.average_duration | round: 1 }}</td>
<td>{{ result.average_prompt_tokens | round: 0 }} prompt + {{ result.average_completion_tokens | round: 0 }} completion</td>
<td>{{ result.date }}</td>
</tr>
{% endfor %}
</tbody>
</table>

## About BC-Bench

TODO

For more information, visit the [BC-Bench repository](https://github.com/microsoft/BC-Bench).