Skip to content

feat: add localhost reliability workspace - #19

Merged
BLCCoreStudio merged 29 commits into
mainfrom
feat/reliability-workspace
Sep 3, 2026
Merged

feat: add localhost reliability workspace#19
BLCCoreStudio merged 29 commits into
mainfrom
feat/reliability-workspace

Conversation

@BLCCoreStudio

Copy link
Copy Markdown
Owner

Summary

  • add a localhost /reliability workspace over the existing reliability.py population-reliability core
  • derive same-stimulus candidate task groups from canonical evaluation JSON plus trusted workflow attribution
  • add a shared read-only workspace evaluation/attribution layer instead of duplicating artifact-loading rules
  • revalidate every selected filename server-side before building PopulationReliabilitySpec
  • reject missing or mismatched workflow attribution, duplicate evaluator identities, duplicate filenames, cross-task artifact reuse, and undersized datasets
  • render Krippendorff alpha, Fleiss kappa, ICC(A,1), applicability reasons, and assumptions without browser-side statistical formulas or universal quality thresholds
  • keep browser reliability reports ephemeral by default with explicit JSON export only
  • add shared responsive navigation across Workbench, Calibration, Review Queue, and Reliability
  • bump package/docs to 0.13.0

Trust / interpretation boundary

  • evaluator identity is accepted only from a valid matching local workflow sidecar
  • workflow artifact_id and task_id are cross-checked against the selected evaluation
  • browser compatibility/dataset keys are UI aids only; they are never trusted as server-side proof
  • the browser does not implement alpha, kappa, or ICC calculations
  • reliability output does not become workflow state, review-queue state, a hidden database, an evaluator leaderboard, or a pass/fail decision
  • no reliability/ or reliabilities/ workspace directory is created by analysis

Validation before PR

  • Ruff: pass
  • strict mypy: pass
  • full tests: pass on Python 3.11, 3.12, and 3.13
  • wheel/package-data smoke: pass
  • installed CLI smoke: pass
  • real localhost HTTP/persistence flow: pass
  • existing desktop/mobile Chromium workbench E2E: pass
  • 390px navigation overflow regression fixed and verified by the existing Chromium gate

Reliability Workspace-specific coverage

  • workspace-generated report is byte-for-structure equivalent to the existing reliability core report
  • candidate grouping requires trustworthy evaluator attribution and unique evaluator IDs
  • malformed workflow task attribution remains visible as untrusted instead of silently disappearing
  • analysis reloads canonical files and rejects attribution changes after candidate discovery
  • repeated use of one evaluation across task groups is rejected
  • dedicated PR workflow validates backend/core regression, JS syntax, wheel assets, version consistency, real localhost server, desktop/mobile Chromium selection → analysis → render → JSON export, and absence of hidden reliability persistence

@BLCCoreStudio
BLCCoreStudio merged commit 7580b7f into main Sep 3, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant