Skip to content

feat(evals): add Muse Spark 1.3 measured results (2026-09-14) - #91

Merged
ericjuta merged 4 commits into
mainfrom
devin/1789398958-evals-muse-spark
Sep 14, 2026
Merged

ericjuta merged 4 commits into
mainfrom
devin/1789398958-evals-muse-spark

Conversation

@devin-ai-integration

Copy link
Copy Markdown
Contributor

Summary

Adds Muse Spark 1.3 (native muse_cli client, 2026-09-14 campaign) as a third measured model in apps/evals, inline on the existing pages per the PR #89 conventions (counts displayed, graded-denominator sort key, unranked, Accuracy/Cost ).

Artifacts — byte-identical JSON subset of the supplied bundle under apps/evals/src/results/2026-09-14/muse-spark-1.3/ (comparison report, bundle manifest, provenance, campaign results/run plans/manifests, config, verification records). Excluded: baseline/ (identical to the 2026-09-11 files already checked in), genuine-replay.json, .jsonl trials, native traces, logs, source snapshot, PDF. Only the report's muse side is rendered; the sol side duplicates the existing GPT-5.6 Sol data.

Data modelmeasuredRun (a singleton hardcoded to the OMP run) is replaced by per-model campaign metadata:

interface MeasuredCampaign {
  id: "omp-2026-09-11" | "muse-2026-09-14";
  date; harness; repetitions; timeoutMs; sourceCommit;
  executableSourceCommit?; prUrl?; prLabel?; abortedRun?;   // OMP only
  limitations: readonly string[];                             // Muse: report.methodology
}
MeasuredModel.campaign: MeasuredCampaign
export const measuredCampaigns = [ompCampaign, museCampaign];

Muse normalisation (museResult/museCases): total = counts.dispatched, timeouts unscored, cases grouped from trials by caseId (verdict/checks/categories/duration/tokens; timeouts have null checks → {}); Spot cases reuse spotComparison.casePrompts for the shared 4 case ids. Numbers: Spot 9 / 12 (3 timeouts), Perps 42 / 54, Predictions 10 / 39 (1 timeout → sort key 10/38).

Pages — Model profile reads model.campaign.* (PR link / executable commit / aborted-run panel only when present; Limitations whenever the campaign has any). Leaderboard shows one indicator line per campaign and the row's own harness. Task explorer renders a per-model table for Perps/Predictions (was primary-model only); Muse's Spot row shows timeout cells. Methodology renders one block per campaign; the OMP block keeps the PR #85 link and limitations list.

Tests extended in apps/evals/__tests__/measured.test.ts. Typecheck, tests, fmt, lint, build, and check:public-boundary pass locally.

Muse profile

Link to Devin session: https://app.devin.ai/sessions/9924c1a614b94139a2fda346027389e0
Open in Devin Desktop: https://app.devin.ai/desktop/session/9924c1a614b94139a2fda346027389e0?variant=devin
Requested by: @ericjuta

ericjuta and others added 2 commits September 14, 2026 15:20
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

Original prompt from Eric

make a pr from the patches @askgina/plugins in the zip

@vercel

vercel Bot commented Sep 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
plugins-evals Ready Ready Preview Sep 14, 2026 3:35pm UTC

Request Review

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Completed a recorded browser audit of PR #91, including the Methodology copy corrections in fbba9d8 (144b75f is formatting-only). No unresolved issues remain in the requested flows.

Muse campaign — verified UI
  • Muse totals: 9 / 12 Spot, 42 / 54 Perps, 10 / 39 Predictions; measured rows unranked, Accuracy/Cost .
  • Graded-denominator sorting, stable search ranks, campaign-specific harness labels, latency-scatter inclusion / cost-scatter exclusion all behave as intended.
  • Muse profile has its own 2026-09-14 provenance, correct dimensions, 4/18/13 cases, eleven limitations, and no OMP-only metadata (PR Fix native OMP eval catalog compatibility and tool evidence #85 link, executable commit, aborted run).
  • Downloaded JSON has the expected filename and campaign object.
Muse profile Breakdown at 700px
Muse measured profile Muse responsive breakdown
Tasks, methodology, regression, browser health
  • Spot task matrices include Muse with timeout cells; Perps/Predictions show separate dated tables for Sol and Muse.
  • Both Methodology campaign blocks render; two stale-copy issues found during testing were fixed and rechecked.
  • Regression: GPT-5.6 Sol retains PR Fix native OMP eval catalog compatibility and tool evidence #85 link, executable commit, aborted-run panel, and 2026-09-11 footer.
  • Pages fit at 700px; wide tables scroll internally. No console errors; all favicon assets return 200.

@ericjuta
ericjuta merged commit f6e5193 into main Sep 14, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant