Skip to content

feat: add calibration dashboard and history - #11

Merged
BLCCoreStudio merged 16 commits into
mainfrom
feat/calibration-dashboard-history
Sep 2, 2026
Merged

feat: add calibration dashboard and history#11
BLCCoreStudio merged 16 commits into
mainfrom
feat/calibration-dashboard-history

Conversation

@BLCCoreStudio

Copy link
Copy Markdown
Owner

Summary

Adds a localhost Calibration workspace over the existing multi-evaluator calibration core. It operates on actual saved evaluation artifacts plus workflow evaluator IDs, persists append-only calibration reports, and never rewrites original evaluations or workflow sidecars.

Dashboard

  • dedicated /calibration workspace
  • compatibility-grouped candidate selection
  • two-or-more-evaluator calibration
  • scalar, pairwise, and timestamped-audio agreement views
  • evaluator aggregate bars
  • criterion observation/agreement table
  • audio pair agreement with F1, severity agreement, and temporal similarity
  • configurable audio timestamp tolerance
  • JSON export
  • append-only calibration history with reopen/download support

Artifact and attribution boundaries

  • candidates are read from workspace/evaluations/
  • evaluator IDs are read from matching workflow sidecars
  • a missing or corrupt workflow sidecar does not hide a valid evaluation; it is shown as unavailable for calibration because attribution cannot be established safely
  • client-side compatibility grouping is UX only; the server and calibration core revalidate task/type/rubric/source compatibility
  • reports are stored separately under workspace/calibrations/
  • source evaluation artifacts remain byte-for-byte unchanged while generating a calibration report

Semantics

Calibration remains diagnostic. The dashboard does not:

  • infer which evaluator is correct
  • automatically pass/fail/rank evaluators
  • define a universal agreement threshold
  • adjudicate disagreements
  • use an external LLM or telemetry
  • upload evaluation or calibration data

Packaging and CI

  • package version bumped to 0.6.0
  • README and architecture updated to the real browser/dashboard surface
  • dedicated Calibration Dashboard workflow checks dashboard backend tests, JavaScript syntax, and wheel-packaged HTML/CSS/JS assets
  • full feature-branch CI is green on Python 3.11 / 3.12 / 3.13 with Ruff, strict mypy, 77 tests, 91.79% coverage, wheel/CLI smoke checks, live localhost HTTP/persistence checks, and desktop/mobile Chromium E2E regression coverage

Additional robustness fix

During edge-case testing, a corrupt workflow sidecar was found to hide an otherwise valid evaluation from calibration candidates. The backend now isolates those failures: the evaluation remains visible but is marked non-calibration-ready until evaluator attribution is valid.

@BLCCoreStudio
BLCCoreStudio merged commit 7125c3d into main Sep 2, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant