feat(teacher): course analytics page + confusion-hotspots widget - #579
Merged
Conversation
Teachers had no way to see how an exercise performs across students. The
data existed — `lms_get_confusion_hotspots` already fused practice, exercise
and exam-question outcomes with Elo `item_ratings` — but it was reachable
only from an MCP client, had no widget, and nothing in the web app read it.
Adds both surfaces, and closes the gap that actually answers "is this too
hard?": `exercises.difficulty_level` (the teacher's own easy/medium/hard
label) was never compared against the rating the platform measured from real
attempts. Both surfaces now show the label beside the measured rating and
flag the divergence.
Web
- /dashboard/teacher/courses/[id]/analytics — server-rendered, 7/30/90-day
window via plain links (no client JS), linked from the course header.
- lib/analytics/confusion-hotspots.ts fuses four signals: practice drills,
exercise evaluations, lesson checkpoints and exam-question miss rates.
`lesson_checkpoint_attempts` was never read by any teacher surface
despite shipping a purpose-built `..._teacher_metrics_idx`.
- Sweeps use `fetchAllRows`, so a course busy enough to hit PostgREST's
1000-row cap cannot silently under-report. A signal that fails to load
becomes a named warning in the UI rather than an empty list that looks
like "nobody is struggling".
- Ownership-gated to the course author; every read runs under the teacher's
own RLS policies, no admin client.
MCP
- resources/confusion-hotspots/widget.tsx, wired to the existing tool.
- hardest_items now carries difficulty_level + mismatch.
- Hotspot scope "lesson" renamed to "practice": those rows are drill
topics, which may not map to a lesson at all.
- Demo fixtures (default / all-clear / empty) for no-DB previews.
Typing the Supabase client (rather than `SupabaseClient<any>`) caught two
real defects in the hand-written row shapes: `checkpoint_id` is a number,
and `submission_date` is nullable and so cannot be compared bare when
picking a student's latest retake.
Verified: Next build + typecheck + lint clean; mcp-server build (19 widgets)
and type check pass; widget rendered headlessly in all three fixture states;
page rendered live against seeded local data as creator@codeacademy.com and
the seed rows removed afterwards.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VqQTE7LDYPx7d15BcWLGfC
The manual pass could only exercise practice and exercise hotspots with a single student, because every source table was empty locally. Exam questions, lesson checkpoints, the fetchAllRows paging loop and the warning path all shipped unverified. 23 tests close that. Each test was mutation-checked — the suite was re-run against a deliberately broken copy of the logic to prove the assertions can fail: - first-attempt-wins instead of latest-attempt → caught - miss threshold moved from 70% to 50% → caught - `easy` given a floor (so it could read easier) → caught - oldest retake wins instead of newest → caught That last mutation initially SURVIVED, which exposed a genuinely misleading test: a NULL `submission_date` never passes the `.gte(cutoff)` window filter (NULL comparisons are never true), so such a row cannot reach the retake logic at all. The test claiming to prove null-handling was passing for the wrong reason. Split into an honest pair: one that exercises retake selection with two dated submissions, and one that pins the real upstream behaviour. fake-supabase gains `range()` (needed by fetchAllRows) and a real `order()`. `count` now reports the FULL match count rather than the post-limit count, matching PostgREST — a per-page count would make a truncated sweep look complete, defeating the assertion fetchAllRows exists to make. All 611 pre-existing tests still pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VqQTE7LDYPx7d15BcWLGfC
The analytics page shipped its per-hotspot "why" line as a prebuilt English sentence built inside lib/analytics/confusion-hotspots.ts, so on /es every heading, badge and footer translated while the line carrying the actual finding stayed in English: 4 of 5 student(s) still failing on their latest attempt · average 52% 3 of 5 student(s) missed it on their latest submission · 60% miss rate Caught by rendering the page under /es for the QA screenshots, not by a test — the strings were correct, just not translatable. Hotspot now carries numbers only (`totalAttempts` replaces `evidence`) and the page formats them through next-intl. Each signal counts something different, so each scope gets its own message rather than one generic sentence that would read wrong for three scopes out of four. The MCP tool keeps its English `evidence` string on purpose: its reader is a model, not a localised UI. Noted in the twin-implementation header. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VqQTE7LDYPx7d15BcWLGfC
Page rendered against seeded data in the local Code Academy tenant with all four signals populated (practice, exercise, checkpoint, exam question) and both label-vs-Elo mismatch directions present. 7d/30d/90d differ because the seed spreads attempts across three time bands, so the window switcher is shown doing real work rather than re-rendering the same list. Widget shots cover the three demo fixtures: default, all-clear, empty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VqQTE7LDYPx7d15BcWLGfC
guillermoscript
marked this pull request as ready for review
July 28, 2026 01:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Teachers could not see how an exercise or AI-generated drill performs across students. The analysis already existed —
lms_get_confusion_hotspotsfuses practice, exercise and exam-question outcomes with Eloitem_ratings— but it was reachable only from an MCP client, had no widget, and nothing underapp/read it.This adds both surfaces and closes the gap that actually answers "is this too hard?":
exercises.difficulty_level(the teacher's own easy/medium/hard label) was never compared against the rating the platform measured from real attempts.Web —
/dashboard/teacher/courses/[id]/analyticsHarder than labelled/Easier than labelledflag.lesson_checkpoint_attemptswas never read by any teacher surface despite shipping a purpose-built..._teacher_metrics_idx.fetchAllRows, so a course busy enough to hit PostgREST's 1000-row cap cannot silently under-report. A signal that fails to load surfaces as a named warning instead of an empty list that reads as "nobody is struggling".MCP
resources/confusion-hotspots/widget.tsx, wired to the existing tool — it was the only analytics tool without a widget.hardest_itemsnow carriesdifficulty_level+mismatch."lesson"→"practice": those rows are drill topics that may not map to a lesson at all.default/all-clear/empty) for no-DB previews.Notes for the reviewer
packages/*workspace, so the two cannot share code; the shared thresholds (severity formula,MISS_RATIO,LOW_SCORE,DIFFICULTY_BANDS) are kept identical and documented in both files.SupabaseClient<any>) caught two real defects:checkpoint_idis a number, andsubmission_dateis nullable and so cannot be compared bare when picking a student's latest retake.QA
npm run dev, sign in ascreator@codeacademy.com/password123oncode-academy.lvh.me:<port>.item_ratings/practice_attempts/exercise_evaluationsfor the course to see calibration + hotspots.cd mcp-server && MCP_DEMO_WIDGETS=1 npm run start, then renderlms_demo_confusion_hotspots(variantsdefault,all-clear,empty).Verification performed
build+typecheck+lintclean; 634 unit tests pass (23 new, every one mutation-checked).useCallTooltype errors in other widgets are untouched by this PR.Screenshots
Analytics page — 30d (English)
Four scopes present (Exercise, Checkpoint, Practice, Exam question), 3 items flagged as mislabelled, both "Harder than labelled" and "Easier than labelled".
The look-back window actually filters
The seed spreads attempts over three time bands, so the three windows return different data — 7d sees 10 practice / 20 exercise / 5 checkpoint / 3 exam, 90d sees 17 / 27 / 9 / 5.
Spanish
This render caught a real bug, now fixed in
a0452bb: the hotspot detail lines were a prebuilt English sentence, so they stayed English while the rest of the page translated.Widget (MCP)
Not covered
No Playwright E2E. Non-author access is refused by the same
author_idownership check the rest of the teacher area uses, but is not exercised by a test. Elo ratings in the screenshots were back-filled to the values a live item reaches after months of traffic; attempt counts and pass/fail outcomes come from the seeded attempts themselves.🤖 Generated with Claude Code
https://claude.ai/code/session_01VqQTE7LDYPx7d15BcWLGfC