[evals] Daily Evals Feature Report - 2026-07-26

[github-actions]

Executive Summary

Across 16 workflows and 18 runs with evals results, the evals pipeline was healthy operationally but degraded in behavior quality. No evals job failures were observed, however the overall YES rate was 40.5%, below the healthy threshold.
12 eval rows were UNKNOWN; I counted them as non-YES for the pass analysis. Only 3/18 runs passed every question, with failures concentrated in Issue Monster and several single-run workflows.

Note

Status: DEGRADED - evals jobs produced results on every analyzed run, but the overall YES rate stayed below 60%.

Key Metrics

Metric	Value
Workflows with evals	16
Runs analyzed	18
Runs with evals results	18
Evals job success rate	100.0%
Overall YES rate	40.5%

Per-Workflow Pass Rates

Workflow	Runs	Evals Job Success	Run Pass Rate	Lowest-Scoring Question
Artifacts Usage Report	1	100.0%	0.0%	"Was a comprehensive summary report of artifacts usage produced?" (0.0% YES)
CLI Version Checker	1	100.0%	0.0%	"Did the agent check for new versions of agentic CLI tools (Claude Code, GitHub Copilot CLI, Codex, MCP servers, etc.)?" (0.0% YES)
Code Scanning Fixer	1	100.0%	0.0%	"Was a pull request created with a remediation for a code scanning alert, or was noop used when no fixable alerts existed?" (0.0% YES)
Contribution Check	1	100.0%	0.0%	"Did the agent dispatch PRs to the contribution-checker subagent and produce evaluation results for each PR?" (0.0% YES)
Copilot Session Insights	1	100.0%	0.0%	"Was a report produced with usage patterns, success rates, and performance metrics?" (0.0% YES)
Daily Cli Tools Tester	1	100.0%	0.0%	"Did the agent run exploratory tests on the audit, logs, and compile CLI tools?" (0.0% YES)
Daily Hippo Learn	1	100.0%	0.0%	"Did the agent run hippo-memory's learn and sleep commands on recent git commits?" (0.0% YES)
Daily Safe Outputs Conformance Checker	1	100.0%	0.0%	"Did the agent run a conformance check against the Safe Outputs specification implementation?" (0.0% YES)
Daily VulnHunter Scan	1	100.0%	0.0%	"Was a security issue created for verified exploitable findings, or was noop used when VulnHunter found nothing actionable?" (0.0% YES)
Issue Arborist	1	100.0%	0.0%	"Did the agent analyze recent issues and identify related issue relationships?" (0.0% YES)
Issue Monster	3	100.0%	0.0%	"Does the agent output show that at most one issue was assigned to Copilot per run?" (0.0% YES)
Tidy	1	100.0%	0.0%	"Did the agent run code formatting and tidying tools on the codebase?" (0.0% YES)
[aw] Failure Investigator (6h)	1	100.0%	0.0%	"Did the agent investigate agentic workflow failures from the last 6 hours and produce findings?" (0.0% YES)
Auto-Triage Issues	1	100.0%	100.0%	"Did the agent apply at least one label to an unlabeled issue, or correctly call noop when no unlabeled issues were found?" (100.0% YES)
Daily Semgrep Scan	1	100.0%	100.0%	"Was a code scanning alert created for real security findings, or does the agent output confirm no vulnerabilities were found?" (100.0% YES)
Sub-Issue Closer	1	100.0%	100.0%	"Did the agent check parent issues for the completion status of all their sub-issues?" (100.0% YES)

Per-Question Breakdown per Workflow

Artifacts Usage Report

Question ID	Question	YES	NO	YES Rate
summary_produced	Was a comprehensive summary report of artifacts usage produced?	0	1	0.0%
artifacts_analyzed	Did the agent analyze GitHub Actions artifacts usage across all workflows in the repository?	1	0	100.0%

CLI Version Checker

Question ID	Question	YES	NO	YES Rate
cli_versions_checked	Did the agent check for new versions of agentic CLI tools (Claude Code, GitHub Copilot CLI, Codex, MCP servers, etc.)?	0	1	0.0%
docker_images_checked	Did the agent check for new versions and digest changes of Docker images in pkg/cli/docker_images.go (actionlint, syft, grype, grant, zizmor, poutine, runner-guard, yamllint)?	0	1	0.0%
updates_applied_or_noop	Were version or digest updates applied and a PR created, or was noop used when all tools were already up to date?	0	1	0.0%

Code Scanning Fixer

Question ID	Question	YES	NO	YES Rate
pr_created_or_noop	Was a pull request created with a remediation for a code scanning alert, or was noop used when no fixable alerts existed?	0	1	0.0%
alerts_analyzed	Did the agent analyze code scanning alerts and identify at least one fixable alert, or correctly skip when no fixable alerts were found?	1	0	100.0%

Contribution Check

Question ID	Question	YES	NO	YES Rate
prs-evaluated	Did the agent dispatch PRs to the contribution-checker subagent and produce evaluation results for each PR?	0	1	0.0%
report-created	Was a report issue created summarizing PR compliance with the contributing guidelines?	0	1	0.0%

Copilot Session Insights

Question ID	Question	YES	NO	YES Rate
insights_report_produced	Was a report produced with usage patterns, success rates, and performance metrics?	0	1	0.0%
sessions_analyzed	Did the agent analyze GitHub Copilot coding agent sessions?	0	1	0.0%

Daily Cli Tools Tester

Question ID	Question	YES	NO	YES Rate
cli_tools_tested	Did the agent run exploratory tests on the audit, logs, and compile CLI tools?	0	1	0.0%
test_results_reported	Were the test results reported with any issues or anomalies identified?	1	0	100.0%

Daily Hippo Learn

Question ID	Question	YES	NO	YES Rate
hippo_learn_executed	Did the agent run hippo-memory's learn and sleep commands on recent git commits?	0	1	0.0%
improvements_suggested	Were actionable improvement suggestions produced from the consolidated memory store?	1	0	100.0%

Daily Safe Outputs Conformance Checker

Question ID	Question	YES	NO	YES Rate
conformance_checked	Did the agent run a conformance check against the Safe Outputs specification implementation?	0	1	0.0%
tasks_created_or_noop	Were agentic tasks created for Copilot to address critical/high/medium/low issues, or was noop used when implementation was compliant?	0	1	0.0%

Daily VulnHunter Scan

Question ID	Question	YES	NO	YES Rate
issue_created_or_noop	Was a security issue created for verified exploitable findings, or was noop used when VulnHunter found nothing actionable?	0	1	0.0%
scan_completed	Did the agent download the prepared VulnHunter bundle artifact, load its vulnhunt skill instructions, and complete a repository scan?	1	0	100.0%

Issue Arborist

Question ID	Question	YES	NO	YES Rate
issues_analyzed	Did the agent analyze recent issues and identify related issue relationships?	0	1	0.0%
sub_issues_linked_or_noop	Were related issues linked as sub-issues, or was noop used when no linkable relationships were found?	0	1	0.0%

Issue Monster

Question ID	Question	YES	NO	YES Rate
single_issue_scoped	Does the agent output show that at most one issue was assigned to Copilot per run?	0	3	0.0%
issue_assigned	Did the agent assign at least one issue to the Copilot coding agent, or correctly skip when no suitable issues were found?	3	0	100.0%

Tidy

Question ID	Question	YES	NO	YES Rate
tidy_completed	Did the agent run code formatting and tidying tools on the codebase?	0	1	0.0%
pr_created_or_noop	Was a pull request created with formatting and tidying fixes, or was noop used when no changes were needed?	1	0	100.0%

[aw] Failure Investigator (6h)

Question ID	Question	YES	NO	YES Rate
failures_investigated	Did the agent investigate agentic workflow failures from the last 6 hours and produce findings?	0	1	0.0%
issues_created_or_closed	Were fix sub-issues created for unresolved failures, or were resolved tracking issues closed?	0	1	0.0%

Auto-Triage Issues

Question ID	Question	YES	NO	YES Rate
labels-applied	Did the agent apply at least one label to an unlabeled issue, or correctly call noop when no unlabeled issues were found?	1	0	100.0%
report-created	Was a summary discussion created listing the issues processed and the labels applied?	1	0	100.0%

Daily Semgrep Scan

Question ID	Question	YES	NO	YES Rate
alert_created_or_noop	Was a code scanning alert created for real security findings, or does the agent output confirm no vulnerabilities were found?	1	0	100.0%
scan_completed	Did the agent complete a Semgrep security scan and report on the findings?	1	0	100.0%

Sub-Issue Closer

Question ID	Question	YES	NO	YES Rate
issues_checked	Did the agent check parent issues for the completion status of all their sub-issues?	1	0	100.0%
issues_closed_or_noop	Were completed parent issues closed with a comment, or does the agent output confirm no issues were ready to close?	1	0	100.0%

Quality Signals

• single_issue_scoped: 0.0% YES (0/3) across Issue Monster
• cli_tools_tested: 0.0% YES (0/1) across Daily Cli Tools Tester
• cli_versions_checked: 0.0% YES (0/1) across CLI Version Checker

Recommendations

• Focus prompt and eval alignment work on Issue Monster, CLI Version Checker, and Daily Cli Tools Tester, which contributed the clearest failures.
• Review the single_issue_scoped criterion in Issue Monster; it failed in all three sampled runs, so either the workflow behavior needs tightening or the eval is too strict.
• Keep the evals pipeline as-is operationally; there were no artifact/job failures, so the main issue is agent output quality rather than eval infrastructure.

References

• §30193349325
• §30193138437
• §30192857021

Generated by 🧪 Daily Evals Feature Report · gpt54 · 17.3 AIC · ⌖ 3.33 AIC · ⊞ 11K · ◷

• expires on Aug 2, 2026, 12:06 AM UTC-08:00