Skip to content

Commit 1dfe2b8

Browse files
author
Harald Kirschner
committed
fix: skip posting eval PR comment when no results produced
When the eval step crashes (e.g., missing COPILOT_TOKEN), the results file isn't created. Previously the report step would post a confusing ENOENT error comment to the PR. Now it checks if the results file exists first and silently skips the PR comment, only logging a warning to the Actions run.
1 parent 2905341 commit 1dfe2b8

1 file changed

Lines changed: 13 additions & 3 deletions

File tree

.github/workflows/eval.yml

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -100,11 +100,21 @@ jobs:
100100
const fs = require('fs');
101101
102102
let summary = '';
103+
let hasResults = false;
103104
let isPR = context.eventName === 'pull_request';
104105
106+
// Skip reporting when the eval step crashed (no results produced)
107+
if (!fs.existsSync('.agentrc/evals/results.json')) {
108+
const evalOutcome = '${{ steps.eval.outcome }}';
109+
core.warning(`Eval step finished with outcome "${evalOutcome}" — no results file produced.`);
110+
// Don't post an unhelpful comment to the PR
111+
return;
112+
}
113+
105114
try {
106115
const raw = fs.readFileSync('.agentrc/evals/results.json', 'utf8');
107116
const data = JSON.parse(raw);
117+
hasResults = true;
108118
const results = data.results || [];
109119
const total = results.length;
110120
const passed = results.filter(r => r.verdict === 'pass').length;
@@ -149,14 +159,14 @@ jobs:
149159
summary += `\n</details>\n\n`;
150160
}
151161
} catch (err) {
152-
summary += `## ⚠️ AgentRC Eval\n\nCould not read eval results: ${err.message}\n`;
162+
summary += `## ⚠️ AgentRC Eval\n\nFailed to parse eval results: ${err.message}\n`;
153163
}
154164
155165
// Write to Actions job summary (visible in run UI and PR checks tab)
156166
await core.summary.addRaw(summary).write();
157167
158-
// Also post/update as PR comment
159-
if (isPR) {
168+
// Only post/update PR comment when we have actual results
169+
if (isPR && hasResults) {
160170
const marker = '<!-- agentrc-eval-results -->';
161171
const body = marker + '\n' + summary;
162172

0 commit comments

Comments
 (0)