Preference-aware cross-validation (#5219)

Summary:
Pull Request resolved: #5219

## Summary

When `CrossValidationPlot` encounters preference metrics (e.g., `pairwise_pref_query` backed by PairwiseGP), it switches to a preference-appropriate model quality evaluation:

**Regression metrics** (unchanged): predicted-vs-observed scatter + R² — "are predictions accurate?"

**Preference metrics** (new): pairwise classification accuracy — "does the model correctly predict which arm is preferred?"

Both answer the same question ("is the model trustworthy?") with the appropriate methodology.

### Adapter layer: pair-aware fold splitting

`_pairwise_kfold_train_test_split` in `ax/adapter/cross_validation.py` splits by trial_index instead of arm_name. Each pairwise trial contains exactly two arms forming a comparison pair — splitting by trial keeps pairs intact.

Only trials with pairwise data are used as fold boundaries. Non-pairwise trials (e.g., the BO trial with tracking metrics) remain in every training fold so the full ModelList can be refitted — all metrics need data in every fold.

`compute_pairwise_accuracy` computes the fraction of held-out comparison pairs where the model correctly predicts which arm is preferred (random baseline = 50%).

### Analysis layer: visualization switching

`CrossValidationPlot` accepts `preference_metrics: set[str] | None`. When the current metric is in this set, it:
1. Uses `_pairwise_kfold_train_test_split` as the `fold_generator`
2. Computes classification accuracy via `compute_pairwise_accuracy`
3. Renders an accuracy bar chart (scatter + R² is meaningless when observed = binary 0/1 and predicted = latent utility on an incommensurate scale)

`DiagnosticAnalysis` no longer filters preference metrics from CV. Instead, it passes `preference_metrics` to `CrossValidationPlot` so it can switch visualization mode per metric.

### Additional fixes in this diff

- **Pairwise-aware fold splitting in `best_point.py`**: `get_best_parameters_from_model_predictions_with_trial_index` calls `cross_validate()` internally (for model fit assessment). Previously it used default arm-based fold splitting, which broke pairwise comparison pairs apart — a fold with an odd number of pairwise observations crashes `prep_pairwise_data(reshape to [-1, 2])`. Now passes `_pairwise_kfold_train_test_split` as fold generator when preference metrics are present.

### Compatibility with SAAS + PairwiseGP ModelLists

A fully-Bayesian SAAS outcome model combined with a PairwiseGP in a `ModelList` yields posteriors with an extra leading MCMC-sample batch dimension. This is already handled in `predict_from_model` (landed in D99037272), which the CV path reaches via `BoTorchGenerator.cross_validate -> Surrogate.predict`, so no change in `torch.py` is required. This diff adds an end-to-end regression test (`test_pairwise_cv_with_saas_pairwise_modellist`) that fits a real SAAS + PairwiseGP `ModelList` and runs preference-aware cross-validation — a path that previously had no coverage.

Differential Revision: D99151833

fbshipit-source-id: 9e7a0df0521ab556ddb7adbb9f66cecf47aed4b4

Author: ItsMrLin

ax/adapter/cross_validation.py

@@ -656,6 +656,145 @@ def _kfold_train_test_split(
         )
 
 
+def _pairwise_kfold_train_test_split(
+    folds: int,
+    training_data: ExperimentData,
+    preference_metric_name: str | None = None,
+) -> Iterable[CVData]:
+    """Return train/test CV splits based on trial indices.
+
+    For pairwise/preference data, each comparison trial contains exactly two
+    arms forming a pair. Splitting by trial_index (instead of arm_name) keeps
+    comparison pairs intact -- both arms of a pair are always together in
+    the same fold.
+
+    When ``preference_metric_name`` is provided, only trials with observations
+    for that metric are used as fold boundaries. This prevents holding out
+    trials that have no pairwise data (e.g., a BO trial with only tracking
+    metrics), which would remove all data for those metrics and cause
+    downstream model-fitting failures.
+
+    Args:
+        folds: Number of folds. Use -1 for leave-one-out CV (one trial per fold).
+        training_data: Training data to split.
+        preference_metric_name: If provided, only split on trials that have
+            non-NaN observations for this metric.
+
+    Returns:
+        Yields CVData with train/test splits.
+    """
+    if preference_metric_name is not None:
+        obs_mean = training_data.observation_data["mean"]
+        if preference_metric_name in obs_mean.columns:
+            non_null = obs_mean[preference_metric_name].dropna()
+            trial_indices = sorted(set(non_null.index.get_level_values("trial_index")))
+        else:
+            trial_indices = sorted(
+                set(training_data.arm_data.index.get_level_values("trial_index"))
+            )
+    else:
+        trial_indices = sorted(
+            set(training_data.arm_data.index.get_level_values("trial_index"))
+        )
+    n = len(trial_indices)
+    if n < 2:
+        raise UnsupportedError(
+            "Pairwise cross validation requires at least two trials in the "
+            f"training data. Only {n} trials were found."
+        )
+    elif folds > n:
+        raise ValueError(
+            f"Training data only has {n} trials, which is less than {folds} folds."
+        )
+    elif folds < 2 and folds != -1:
+        raise ValueError("Folds must be -1 for LOO, or > 1.")
+    elif folds == -1:
+        folds = n
+
+    trial_arr = np.array(trial_indices)
+    if folds != n:
+        np.random.shuffle(trial_arr)
+    test_size = n // folds
+    final_size = test_size + (n - folds * test_size)
+    for fold in range(folds):
+        trial_arr = np.roll(trial_arr, test_size)
+        n_test = test_size if fold < folds - 1 else final_size
+        train_trials = set(trial_arr[:-n_test].tolist())
+        test_trials_set = set(trial_arr[-n_test:].tolist())
+        # Non-split trials (e.g., BO trials with only tracking metrics)
+        # must stay in every training fold so the full ModelList can be
+        # refitted (all metrics need data).
+        # TODO(D94970662): Once Experiment._trial_type_to_metric_names
+        # lands, use experiment.trials_for_type() instead of data-driven
+        # inference to determine split-eligible vs always-train trials.
+        all_trials = set(training_data.arm_data.index.get_level_values("trial_index"))
+        train_trials = train_trials | (all_trials - set(trial_indices))
+        yield CVData(
+            training_data=training_data.filter_by_trial_index(
+                trial_indices=train_trials
+            ),
+            test_data=training_data.filter_by_trial_index(
+                trial_indices=test_trials_set
+            ),
+        )
+
+
+def compute_pairwise_accuracy(
+    cv_results: list[CVResult],
+    metric_name: str,
+) -> float:
+    """Compute classification accuracy for pairwise preference CV.
+
+    For each held-out comparison pair (two arms from the same trial), checks
+    whether the model correctly predicts which arm is preferred (i.e., the arm
+    with higher predicted utility matches the arm with observed label = 1).
+
+    Args:
+        cv_results: Cross-validation results from cross_validate().
+        metric_name: The preference metric name to evaluate.
+
+    Returns:
+        Fraction of correctly predicted comparisons (random baseline = 0.5).
+    """
+    # Group CV results by trial_index to find comparison pairs.
+    trial_groups: dict[int | None, list[CVResult]] = defaultdict(list)
+    for result in cv_results:
+        trial_groups[result.observed.features.trial_index].append(result)
+
+    correct = 0
+    total = 0
+    for _trial_idx, results in trial_groups.items():
+        if len(results) != 2:
+            continue
+        r0, r1 = results
+
+        # Find the metric index in the observation data.
+        try:
+            idx0 = list(r0.observed.data.metric_signatures).index(metric_name)
+            idx1 = list(r1.observed.data.metric_signatures).index(metric_name)
+            pred_idx0 = list(r0.predicted.metric_signatures).index(metric_name)
+            pred_idx1 = list(r1.predicted.metric_signatures).index(metric_name)
+        except ValueError:
+            continue
+
+        obs0 = r0.observed.data.means[idx0]
+        obs1 = r1.observed.data.means[idx1]
+        pred0 = r0.predicted.means[pred_idx0]
+        pred1 = r1.predicted.means[pred_idx1]
+
+        # The arm with observed label = 1 is preferred. Check if the model
+        # predicts higher utility for the preferred arm.
+        if obs0 == obs1:
+            continue  # Skip ties in observed labels
+        preferred_is_0 = obs0 > obs1
+        model_predicts_0 = pred0 > pred1
+        if preferred_is_0 == model_predicts_0:
+            correct += 1
+        total += 1
+
+    return correct / total if total > 0 else 0.0
+
+
 def gen_trial_split(
     training_data: ExperimentData,
     test_trials: list[int],