Train a cross-validated run through the units, unless it is nested

FitModelOverFoldsUnit is FitModelUnit's sibling: it takes a list of partition
sets instead of one, which is a different REQUIRES and so a different unit.
Everything around the fit is the shared mixin; what is written here is the
objective the search measures -- the whole fold loop, so one trial costs k fits
-- and what happens once the search is over.

ModelJob composes it for a fold run that is not nested. Nested cross-validation
still trains through the strategy: its inner splitter is a required component
field, and a component field cannot be made optional without leaving the user
without a selector, so it is a further sibling rather than a flag on this one.

Three decisions worth naming.

The per-fold scores are published rather than aggregated in the unit. A summary
row carries a standard deviation, and a unit may not write domain rows -- the
one sanctioned write in the domain layer has nowhere to put one. So the unit
hands the numbers over and the job does the arithmetic and the writing, where
every other row it persists is written. A single fold gets a deviation of zero
rather than none, because none is what the reserved-rows measurement carries
and the two say different things.

A trial records one row per split holding the mean over its folds, not one per
fold: the folds of a trial measure a hyperparameter setting rather than the
model that gets kept, and recording each would bury the rows that describe it.
That write is guarded on the run, which _save_metrics does not guard for
itself -- a caller with no run would write rows against a foreign key pointing
at nothing, and they insert without complaint because nothing enforces it.

Scoring the reserved rows is not the unit's. It is an ordinary LAST metric, so
it is EvaluateModelUnit, the same one a holdout run uses, and whether there is
anything to score is the caller's to know: a session that reserved nothing
leaves that partition empty rather than absent.

calculate_metrics now returns what it wrote, so a caller that wants both the
row and the number scores the split once instead of twice.

The assertion that the optimizer gave back the model it was handed caught a
real gap while this was written: with the data attached at fit time rather than
at build time, nothing had pointed the model at anything during a fold search.
It does now, per fold, the same as the scoring loop.

1001 passed across units, dag, spike, api and evaluation; both nets unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: Felipedino

DashAI/back/initial_components.py

@@ -524,6 +524,7 @@
 )
 from DashAI.back.units.evaluate_model_unit import EvaluateModelUnit
 from DashAI.back.units.fit_converter_unit import FitConverterUnit
+from DashAI.back.units.fit_model_over_folds_unit import FitModelOverFoldsUnit
 from DashAI.back.units.fit_model_unit import FitModelUnit
 from DashAI.back.units.generate_global_explanation_unit import (
     GenerateGlobalExplanationUnit,
@@ -732,6 +733,7 @@ def get_initial_components():
         PrepareAndFoldUnit,
         BuildModelUnit,
         FitModelUnit,
+        FitModelOverFoldsUnit,
         EvaluateModelUnit,
         EvaluateModelToArtifactUnit,
         SaveModelUnit,

DashAI/back/job/model_job.py

@@ -5,14 +5,21 @@
 from sqlalchemy import exc
 from sqlalchemy.orm.attributes import flag_modified
 
-from DashAI.back.dependencies.database.models import Dataset, ModelSession, Run
+from DashAI.back.core.enums.metrics import LevelEnum, SplitEnum
+from DashAI.back.dependencies.database.models import (
+    Dataset,
+    Metric,
+    ModelSession,
+    Run,
+)
 from DashAI.back.evaluation.base_evaluation_strategy import BaseEvaluationStrategy
 from DashAI.back.job.base_job import BaseJob, JobError
 from DashAI.back.optimizers.base_optimizer import BaseOptimizer
 from DashAI.back.splitters.splits_payload import normalize_splits_payload
 from DashAI.back.units.build_model_unit import BuildModelUnit
 from DashAI.back.units.context import ExecutionContext
 from DashAI.back.units.evaluate_model_unit import EvaluateModelUnit
+from DashAI.back.units.fit_model_over_folds_unit import FitModelOverFoldsUnit
 from DashAI.back.units.fit_model_unit import FitModelUnit
 from DashAI.back.units.load_dataset_unit import LoadDatasetUnit
 from DashAI.back.units.prepare_and_fold_unit import PrepareAndFoldUnit
@@ -224,10 +231,47 @@ def run(
 
                         self.report_progress(0.85, "Computing metrics")
                         EvaluateModelUnit(run_id=run_id, splits=scored_splits)(ctx)
+                    elif not run.nested:
+                        fit_folds = FitModelOverFoldsUnit(
+                            optimizer={
+                                "component": run.optimizer_name,
+                                "params": run.optimizer_parameters,
+                            },
+                            goal_metric=run.goal_metric,
+                            run_id=run_id,
+                            artifact_prefix=str(run_id),
+                            scored_splits=[
+                                name for name in scored_splits if name != "TEST"
+                            ],
+                        )
+                        fit_folds(ctx)
+
+                        plot_paths = ctx.require("plot_paths")
+                        if ctx.has("best_parameters"):
+                            run.parameters = ctx.get("best_parameters")
+                            flag_modified(run, "parameters")
+                            db.commit()
+
+                        self.report_progress(0.85, "Computing metrics")
+                        self._aggregate_fold_metrics(
+                            db, run_id, ctx.require("fold_metrics")
+                        )
+
+                        # The rows the session reserved are the only ones no
+                        # fold and no trial ever saw, so they are the only
+                        # honest estimate left once a model is picked out of a
+                        # comparison table -- and scoring them is an ordinary
+                        # LAST metric, so it is the same unit a holdout run
+                        # uses. Whether there is anything to score is the
+                        # caller's to know: a session that reserved nothing
+                        # leaves that partition empty rather than absent.
+                        if len(x[-1]["test"]) > 0:
+                            EvaluateModelUnit(run_id=run_id, splits=["TEST"])(ctx)
                     else:
-                        # Fold runs still train through the strategy. Their loop
-                        # is the next piece to move; everything before and after
-                        # it is already the units'.
+                        # Nested cross-validation still trains through the
+                        # strategy: its inner splitter is a required component
+                        # field, so it is a further sibling unit rather than a
+                        # flag on the one above, and it is not written yet.
                         evaluation_estrategy: BaseEvaluationStrategy = strategy_class(
                             factory=ctx.require("factory"),
                             optimizer=preparation_results["optimizer"],
@@ -295,6 +339,65 @@ def run(
                 ctx.clear_cache()
                 gc.collect()
 
+    @staticmethod
+    def _aggregate_fold_metrics(db, run_id: int, fold_metrics: Dict[str, Any]) -> None:
+        """Summarise the per-fold scores into one row per split and metric.
+
+        The unit that fitted the folds publishes their scores rather than
+        aggregating them, because a summary row carries a standard deviation
+        and a unit may not write domain rows -- the one sanctioned write in the
+        domain layer has nowhere to put one. So the arithmetic and the writing
+        happen here, where every other row this job persists is written.
+
+        A single fold gets a deviation of zero rather than none: none is what
+        the reserved-rows measurement carries, and the two say different
+        things -- "one fold, so nothing varied" against "not a summary at all".
+
+        Parameters
+        ----------
+        db : Session
+            The session this job is already holding.
+        run_id : int
+            The run the rows belong to.
+        fold_metrics : dict
+            ``{split name: {metric name: [one score per fold]}}``.
+        """
+        import numpy as np
+
+        for split_name, by_metric in fold_metrics.items():
+            for metric_name, values in by_metric.items():
+                if not values:
+                    continue
+                existing = (
+                    db.query(Metric)
+                    .filter_by(
+                        run_id=run_id,
+                        split=SplitEnum[split_name],
+                        level=LevelEnum.LAST,
+                        name=metric_name,
+                    )
+                    .first()
+                )
+                mean = float(np.mean(values))
+                deviation = float(np.std(values)) if len(values) > 1 else 0.0
+
+                if existing:
+                    existing.value = mean
+                    existing.std_value = deviation
+                else:
+                    db.add(
+                        Metric(
+                            run_id=run_id,
+                            split=SplitEnum[split_name],
+                            level=LevelEnum.LAST,
+                            name=metric_name,
+                            value=mean,
+                            std_value=deviation,
+                            step=0,
+                        )
+                    )
+        db.commit()
+
     def _prepare_dataset_and_components(
         self, run_id: int, db, component_registry
     ) -> Dict[str, Any]:

DashAI/back/models/base_model.py

@@ -349,6 +349,13 @@ def calculate_metrics(
             labels stored in the model for the given split are used.
             Defaults to None.
 
+        Returns
+        -------
+        Dict[str, float] or None
+            What was written, so a caller that also wants the numbers does not
+            have to score the same split twice. ``None`` when nothing was
+            written: no run to write against, or nothing to score.
+
         Notes
         -----
         A metric row is keyed by the run it belongs to, so a model with no run
@@ -366,11 +373,11 @@ def calculate_metrics(
         # checked for metrics first. Treating "no attribute" as "no run" keeps
         # that path working and is the same answer for any caller that has one.
         if not getattr(self, "run_id", None):
-            return
+            return None
 
         results = self.compute_metrics(split=split, x_data=x_data, y_data=y_data)
         if results is None:
-            return
+            return None
 
         # Save to database
         self._save_metrics(
@@ -392,6 +399,8 @@ def calculate_metrics(
         ):
             self._epoch_reporter(results, log_index)
 
+        return results
+
     def prepare_dataset(
         self, dataset: "DashAIDataset", is_fit: bool = False
     ) -> "DashAIDataset":