Introduce Reports for run evaluation - #868
Merged
Merged
Conversation
ExplainersPlot held the only renderer for the backend artifact wire format: leaf artifacts, grouped selectors, and the layout that pairs a selector with its batch. Any second consumer had to import from the explainers tree. Move that into a shared ArtifactList. The group picker becomes a render prop so the shared component never reaches for the dataset API, which only the explainers' instance table needs.
A report is the artifact-valued sibling of a metric: both compare a model's predictions over one split against the truth, and they differ only in codomain. A metric reduces to a rankable float; a report reduces to a shape a float cannot hold, such as a K x K confusion grid or an ROC curve. The hook takes no features and no model, so a report cannot probe how the model responds to inputs; that stays the explainer's job.
Reports are added from the right sidebar on the run's Reports tab, mirroring how explainers are added: click a compatible report, pick the split it should describe, configure it when it takes parameters, and the creator both stores the row and enqueues its job. The parameter step is dropped for the reports that take none, since a one step wizard is just a dialog with a Save button.
ModelListItem defaults its drag type to the model type, so report rows were draggable but announced themselves as models. The central panel read the drop as a model and opened the Add Model dialog instead of the report creator. Give the rows the report drag type and a payload of just the name and display name, and teach the panel's drag handlers the third type.
The colorscale selector always read and wrote layout.coloraxis.colorscale. That only reaches the plot for figures built by px.imshow, which point their trace at a shared colour axis. A figure built straight from go.Heatmap, like the confusion matrix report, carries its colorscale on the trace, so every edit landed on a layout key nothing renders from and the colours never moved. Follow the same rule the colorbar fields already followed: write to the shared axis only when the trace references one, and to the trace otherwise.
Plotly reverses a scale with a flag rather than by rewriting its stops, so one switch covers a named preset and a hand built array alike. It stays visible in both modes: hiding it in array mode would leave a flag silently reversing a set of stops the user built by hand.
A report's edits were client side only: ArtifactViewer kept them in local state, so a reload showed the computed figure again. Reports already had a plot_overrides column; nothing read or wrote it. Add the save and reset endpoints, apply the stored overrides when the artifacts are read, and have the card seed its overridden set from the response so a reload still offers a reset. The override helper moves from the explainers endpoint module into core.artifacts, since it is artifact logic that two callers now share.
ArtifactViewer held a saved edit only in local state, and cleared it whenever the artifact prop changed. A local explainer's instance picker swaps which artifact sits in each slot, so switching instance dropped the edit and switching back showed the stale computed figure, since the fetched list still carried it. Fold the saved figure into that list instead, so the artifact itself carries the edit for as long as the list lives. Also remove the reset edits button and the plumbing that existed only to drive it: the overridden index bookkeeping, the reset client calls and the label.
Mirrors the explainer card. ReportStatus uses the same codes as the run and explainer statuses, so the shared RunStatusDot maps them without a second colour table: grey not started, blue queued or running, green finished, red failed. The dot reads the polled status rather than the prop, so a card that is still computing turns green in place when the job lands.
Deleting a report threw away a computed job and any plot edits saved against it, with no way back, on a single click of a small icon. Route it through the same confirmation modal the explainer cards use.
Matches how the explainer list is ordered, so adding one from the sidebar appends it in both tabs rather than pushing the list down in one of them.
Opening the tab lands at the bottom, adding a report glides to it, and the new card flashes a ring for four seconds. New rows are detected by diffing ids against the previous fetch rather than by a flag from the creator, so the three second poll that refreshes running reports cannot replay the scroll or the ring.
Adding a report refreshed the list, and the refresh unmounted every card to show a centred spinner before bringing them back. It happened twice per add, once on create and again when the job landed, which read as lag rather than progress. Only the first load shows the spinner now; later fetches swap the rows in place, as the explainer list already did. Also drop the per card artifact poll. A running report has no artifacts to fetch, and the parent already polls the list while any report is running, so the timer only doubled requests during the wait.
Three costs compounded on the reports tab, which unlike the explainer list mounts every card at once. ArtifactRenderer built a fresh layout object on each render, and react-plotly.js diffs by reference, so every render sent Plotly through a full relayout. Starting a drag sets state on SessionVisualization, which re-renders the whole run detail tree, so the drag itself relayouted every plot on screen before the pointer had moved. ArtifactViewer also kept its fullscreen lightbox mounted, so each artifact held two Plot instances forever, for a dialog that opens with no transition. Memoize the layout and the artifact objects, memoize the renderer so an ancestor re-render cannot reach Plotly, and drop keepMounted.
Reports no longer ask which split to describe. The job runs the report once per partition and wraps each result in a group, so the card renders the selector layout global explainers use: partitions on the left, that partition's artifacts on the right. Partitions are now data rather than part of the API contract. The split enum, the split column and the split step are gone, and PARTITIONS in the job is the single place that knows a holdout run has three of them. Teaching DashAI cross validation means yielding k folds there; no report class changes, because none of them know what a partition is. compute() narrows to leaf artifacts for the same reason: the per partition grouping belongs to the job, and a group cannot contain another group. A partition that cannot be computed, such as a split holding a single class, is skipped and named in the output rather than failing the whole report.
Most reports have nothing to configure now that the split is gone, so their dialog was a single Create button standing between the click and the work. Add them outright instead, and keep the dialog for the two that do take parameters. The component list already carries each schema, so the decision needs no extra request. Dropping a report onto the central panel follows the same rule, with the schema carried in the drag payload; the create and enqueue path is shared so both entry points behave identically. The decision deliberately sits in the click and drop handlers rather than in an effect watching the selection: StrictMode double invokes effects, and this one would POST.
Develop moved PlotOverrideBody into core.artifacts, so the reports only duplicate goes.
Partitions come from the run's splitter now, the same set the predict and local explainer flows offer, so a cross validated run reports on its reserved test rows and the rest.
The shared artifact list built only the save handler, so the reset button the viewer already had could never render.
The report job resolved its partitions with run_splits, which offers every partition a splitter declares plus the whole dataset. A task that predicts forward only, such as forecasting, refuses to return a value for dates inside the training window, so asking it to report on the train partition failed the whole job. Resolve partitions through predictable_splits, the same helper the prediction flow uses, passing the run's task and evaluation strategy. It drops the rows the kept model was fitted on for forward-only tasks, so a forecasting report covers only the partitions that lie after the fit.
Forecasting reports (work on any forecaster's numeric output): - ForecastVsActual: truth and forecast as lines over the observation index. - ResidualsOverTime: residual against observation index, revealing drift. - ResidualAutocorrelation: residual autocorrelation by lag with a confidence band and a configurable max lag. Translation reports (work on any translation model's text output): - PerSegmentComparison: reference against translation per segment with sentence level BLEU, highlighting the worst rows. - SegmentScoreDistribution: histogram of the per segment scores. - LengthComparison: reference against translation length scatter, revealing truncation. All six consume only y_true and y_pred, never model internals, and are registered in get_initial_components under their task's COMPATIBLE_COMPONENTS.
Irozuku
added
front
cristian-tamblay
approved these changes
Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces Reports, artifact valued evaluation components computed once per evaluation partition of a run from (y_true, y_pred, class_names) and rendered in the run results. They are the structured, non rankable sibling of Metrics. A metric condenses a shape into a float; a report renders the shape, such as a confusion matrix, an ROC curve, residuals, or per segment scores.
The branch adds the abstract contract (BaseReport), thirteen concrete reports across classification, regression, forecasting and translation, the async ReportJob that computes them per partition, a Report table and REST endpoints, and a full frontend Reports tab with sidebar, cards, inline creator, partition selector, and persisted plot edits. A forward only fix keeps reports from ever asking a model to predict rows it was fitted on.
Type of Change
Check all that apply like this [x]:
Changes (by file)
Backend
DashAI/back/reports/base_report.py: abstract BaseReport contract, compute(y_true, y_pred, class_names) returning a list of artifacts, the REQUIRES_PROBABILITIES gate, and the as_labels and resolve_class_names helpers.DashAI/back/reports/{classification,regression,forecasting,translation}/: thirteen reports, Confusion Matrix, ROC Curve, Precision Recall Curve, Per Class Breakdown, Predicted vs Actual, Residual Plot, Residual Histogram, Forecast vs Actual, Residuals Over Time, Residual Autocorrelation, Per Segment Comparison, Segment Score Distribution, and Length Comparison.DashAI/back/job/report_job.py: ReportJob computes a report over every partition the run exposes, skips partitions a forward only model cannot predict, and reports skipped partitions as text.DashAI/back/api/api_v1/endpoints/reports.pyandschemas/reports_params.py: create, list and delete reports, fetch artifacts, and persist and reset plot overrides.DashAI/back/dependencies/database/models.pyandDashAI/alembic/versions/e6c3b91a7d48andb7e4d2a19c63: Report table and migrations.DashAI/back/initial_components.py: registers every report in the registry.DashAI/back/api/api_v1/api.pyandcore/enums/status.py: router mount and ReportStatus.Frontend
DashAI/front/src/components/models/runResults/ReportResultsTab.jsxandResultsTabsHeader.jsx: Reports tab in the run results.DashAI/front/src/components/reports/: ReportsSidebar, ReportCard, InlineReportCreator, ConfigureReportStep, and createAndRunReport.js.DashAI/front/src/components/shared/ArtifactList.jsxandArtifactGroupSelector.jsx: shared artifact and grouped selector rendering extracted from the explainer plot flow.DashAI/front/src/utils/artifactOverrides.js: persist and reset plot edits.DashAI/front/src/components/notebooks/explorer/plotLayout/ColorscaleSelector.jsx: reverse toggle for heatmap colorscales.DashAI/front/src/utils/i18n/locales/en,es,pt,de,zh/reports.json: new locale bundles.Tests
tests/back/reports/: unit tests per report, inputs, normalization, and type registration.tests/back/api/test_reports_api.py: end to end create, ReportJob, artifacts, and overrides on holdout and cross validated runs.tests/back/api/test_reports_forecasting_api.py: end to end proof that a forward only run reports only on its forecastable partitions.Testing (optional)
Notes (optional)
predictable_splitshelper the prediction flow uses.