Skip to content

code-review.md

Latest commit

 

History

History
124 lines (111 loc) · 5.68 KB

code-review.md

File metadata and controls

124 lines (111 loc) · 5.68 KB
Raw
layout default
title Code Review - BC-Bench
<style> /* Widen this page so the leaderboard table breathes instead of cramming into the narrow column */ .main-content { max-width: 80rem; } .main-content table { display: table; width: 100%; table-layout: auto; } .main-content table th, .main-content table td { padding: 0.4rem 0.6rem; } </style>

Code Review

This category evaluates an agent's ability to review a Business Central (AL) pull request. Given a diff, the agent produces structured review comments, which are scored against an expected (gold) set of findings.

Unlike the pass/fail categories, code review is scored with Precision / Recall / F1 over the matched comments. Expected and generated comments are paired by a globally optimal (one-to-one) assignment on file and line proximity (within a configured line tolerance), and a pair is only counted as matched when an LLM judge confirms the two describe the same underlying issue. Matched comments are additionally scored on how closely the agent's severity classification tracks the expected severity.

Baseline Leaderboard

{% if site.data.code-review.aggregate and site.data.code-review.aggregate.size > 0 %}

{% assign sorted_results = site.data.code-review.aggregate | sort: "f1" | reverse %} {% for agg in sorted_results %} {% if agg.experiment == null %} {% endif %} {% endfor %}
Agent Model Micro F1 (95% CI) Precision Recall Avg Time Ver
{{ agg.agent_name }} {{ agg.model }} {{ agg.f1 | times: 100.0 | round: 1 }}%{% if agg.f1_ci_low %} ({{ agg.f1_ci_low | times: 100.0 | round: 1 }}-{{ agg.f1_ci_high | times: 100.0 | round: 1 }}%){% endif %} {{ agg.precision | times: 100.0 | round: 1 }}% {{ agg.recall | times: 100.0 | round: 1 }}% {{ agg.average_duration | round: 1 }}s {{ agg.benchmark_version }}
{% else %}

No results available yet. Check back soon!

{% endif %}

Experiment Leaderboard

Compares review-knowledge configurations for the same model (see the Baseline Leaderboard above for the plain agent):

  • Inline knowledge (pre-#8700) — the review checklists BCApps shipped inline before adopting BCQuality, injected as custom instructions.
  • BCQuality (live skills) — the agent dynamically consumes the live BCQuality skill tree.

{% assign experiment_rows = site.data.code-review.aggregate | where_exp: "agg", "agg.experiment" %} {% if experiment_rows and experiment_rows.size > 0 %}

{% assign experiment_results = experiment_rows | sort: "f1" | reverse %} {% for agg in experiment_results %} {% endfor %}
Variant Agent Model Micro F1 (95% CI) Macro F1 (95% CI) Precision Recall Avg Time Ver
{%- if agg.experiment.bcquality -%}BCQuality (live skills) {%- elsif agg.experiment.custom_instructions -%}Inline knowledge (pre-#8700) {%- else -%}Other{%- endif -%} {{ agg.agent_name }} {{ agg.model }} {{ agg.f1 | times: 100.0 | round: 1 }}%{% if agg.f1_ci_low %} ({{ agg.f1_ci_low | times: 100.0 | round: 1 }}-{{ agg.f1_ci_high | times: 100.0 | round: 1 }}%){% endif %} {{ agg.macro_f1 | times: 100.0 | round: 1 }}%{% if agg.macro_f1_ci_low %} ({{ agg.macro_f1_ci_low | times: 100.0 | round: 1 }}-{{ agg.macro_f1_ci_high | times: 100.0 | round: 1 }}%){% endif %} {{ agg.precision | times: 100.0 | round: 1 }}% {{ agg.recall | times: 100.0 | round: 1 }}% {{ agg.average_duration | round: 1 }}s {{ agg.benchmark_version }}
{% else %}

No experiment results available yet. Check back soon!

{% endif %}

How metrics are computed

  • Precision — of the comments the agent generated, the fraction that matched an expected finding. Penalizes noisy reviews.
  • Recall — of the expected findings, the fraction the agent caught. Penalizes missed issues.
  • F1 — harmonic mean of precision and recall; balances both equally (the β=1 case of Fβ).
  • Fβ (β=0.5) — precision-leaning F-score; use when false positives are costly (noisy reviews waste reviewer time).
  • Fβ (β=2) — recall-leaning F-score; weights catching issues more than avoiding noise.
  • Severity MAE — mean absolute error between the agent's and the expected severity levels, over matched comments only. Lower is better; 0 means every matched comment got the severity exactly right.
  • Valid output rate — fraction of tasks whose output parsed into a structured review. Failures score zero on every other metric. (Reported per run.)
  • Micro vs. Macro — Micro sums matched/generated/expected across all tasks (tasks with many comments dominate); Macro averages per-task scores (every task counts equally).
  • 95% CI — confidence interval bootstrapped over the per-task F1 scores, so the leaderboard reports sampling uncertainty even for a single run. The micro F1 CI resamples runs; the Macro F1 CI resamples tasks.

← Back to Home