feat(evals): add Muse Spark 1.3 measured results (2026-09-14) - #91
Merged
Merged
Conversation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
Original prompt from Eric
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
Author
|
Completed a recorded browser audit of PR #91, including the Methodology copy corrections in fbba9d8 (144b75f is formatting-only). No unresolved issues remain in the requested flows. Muse campaign — verified UI
Tasks, methodology, regression, browser health
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Muse Spark 1.3 (native
muse_cliclient, 2026-09-14 campaign) as a third measured model inapps/evals, inline on the existing pages per the PR #89 conventions (counts displayed, graded-denominator sort key, unranked, Accuracy/Cost—).Artifacts — byte-identical JSON subset of the supplied bundle under
apps/evals/src/results/2026-09-14/muse-spark-1.3/(comparison report, bundle manifest, provenance, campaign results/run plans/manifests, config, verification records). Excluded:baseline/(identical to the 2026-09-11 files already checked in),genuine-replay.json,.jsonltrials, native traces, logs, source snapshot, PDF. Only the report'smuseside is rendered; thesolside duplicates the existing GPT-5.6 Sol data.Data model —
measuredRun(a singleton hardcoded to the OMP run) is replaced by per-model campaign metadata:Muse normalisation (
museResult/museCases):total = counts.dispatched, timeouts unscored, cases grouped fromtrialsbycaseId(verdict/checks/categories/duration/tokens; timeouts have null checks →{}); Spot cases reusespotComparison.casePromptsfor the shared 4 case ids. Numbers: Spot 9 / 12 (3 timeouts), Perps 42 / 54, Predictions 10 / 39 (1 timeout → sort key 10/38).Pages — Model profile reads
model.campaign.*(PR link / executable commit / aborted-run panel only when present; Limitations whenever the campaign has any). Leaderboard shows one indicator line per campaign and the row's own harness. Task explorer renders a per-model table for Perps/Predictions (was primary-model only); Muse's Spot row showstimeoutcells. Methodology renders one block per campaign; the OMP block keeps the PR #85 link and limitations list.Tests extended in
apps/evals/__tests__/measured.test.ts. Typecheck, tests, fmt, lint, build, andcheck:public-boundarypass locally.Link to Devin session: https://app.devin.ai/sessions/9924c1a614b94139a2fda346027389e0
Open in Devin Desktop: https://app.devin.ai/desktop/session/9924c1a614b94139a2fda346027389e0?variant=devin
Requested by: @ericjuta