diff --git a/docs/cli.md b/docs/cli.md index 9a2f30ce..0e92d3e8 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -44,6 +44,9 @@ winnow config train # Show prediction configuration winnow config predict +# Show compute-features configuration +winnow config compute-features + # Check configuration with overrides winnow config train data_loader=mztab model_output_dir=models/my_model winnow config predict fdr_method=database_grounded fdr_control.fdr_threshold=0.01 @@ -93,6 +96,30 @@ For comprehensive calibrator configuration options, see: - [Configuration guide](configuration.md) - Complete parameter reference - [Calibration API](api/calibration.md#handling-missing-features) - Feature implementation details +### `winnow compute-features` + +Compute calibration features and write one enriched metadata CSV. Uses the same `data_loader`, `dataset` paths and `calibrator.features` stack as training, but does **not** fit the MLP or save a model. + +```bash +# Defaults (configs/compute_features.yaml) +winnow compute-features + +# Paths and output file +winnow compute-features dataset.spectrum_path_or_directory=data/spectra.ipc dataset.predictions_path=data/preds.csv dataset_output_path=results/features.csv + +# De novo spectra (no ground truth): labelled=false; remove retention_time_feature if present +winnow compute-features labelled=false '~calibrator.features.retention_time_feature' +``` + +**Common parameters:** + +- `data_loader`, `dataset.*`: Same as `winnow train` +- `dataset_output_path`: Output CSV path +- `filter_empty_predictions`: Drop empty or invalid predictions (default: true) +- `labelled`: If true (default), spectrum data must include a `sequence` column; runs each feature's `prepare()` (e.g. iRT calibrator). + +Feature selection matches training: override `calibrator.features` or use `~calibrator.features.` to drop entries. See [Configuration guide](configuration.md#compute-features-configuration). + ### `winnow predict` Apply calibration and FDR control to new data using a trained model. By default, uses a pretrained general model from Hugging Face Hub. @@ -154,6 +181,10 @@ For training (`winnow train`), you need: - **Spectral data**: MS/MS spectra and metadata - **Unique identifiers**: Each PSM must have a unique `spectrum_id` in both input files +### Feature export (`winnow compute-features`) + +Same inputs as training for loading spectra and predictions. With default `labelled=true`, the spectrum file must include a `sequence` column (as for training). With `labelled=false`, pure *de novo* inputs are allowed if the configured feature set does not include `retention_time_feature`, which requires labelled data to fit an iRT predictor model. + ### Prediction data For prediction (`winnow predict`), you need: @@ -233,10 +264,16 @@ Prediction produces two CSV files in the `output-folder` directory: - `psm_q_value`: Q-values - `psm_pep`: Posterior error probabilities (non-parametric method only) - `sequence`: Ground truth sequences (if available, for database-ground method) + - `num_matches`: Number of matched residues between the predicted and ground-truth peptide (if labelled) + - `correct`: Whether the predicted sequence is correct (if labelled) - Filtered to only include PSMs passing the FDR threshold This separation allows users to work with metadata and features separately from predictions and error metrics, making downstream analysis more convenient. +### Compute-features output + +A single CSV at `dataset_output_path`: full metadata after feature computation. No calibrated scores, FDR columns or FDR row filtering (unlike `winnow predict`). + ## Example workflows ### Quick start with defaults @@ -328,10 +365,12 @@ View available options: ```bash winnow --help # List all commands winnow train --help # Command-specific help +winnow compute-features --help winnow predict --help winnow config --help # Config command help winnow config train # View resolved training configuration +winnow config compute-features # View resolved feature computation configuration winnow config predict # View resolved prediction configuration ``` diff --git a/docs/configuration.md b/docs/configuration.md index 7525934e..5294f8c2 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -40,6 +40,7 @@ configs/ │ ├── nonparametric.yaml │ └── database_grounded.yaml ├── train.yaml # Main training config +├── compute_features.yaml # Feature-only export (no MLP fit) ├── calibrator.yaml # Model architecture and features └── predict.yaml # Main prediction config ``` @@ -64,6 +65,9 @@ winnow train model_output_dir=models/my_model # Change multiple parameters winnow predict data_loader=mztab fdr_control.fdr_threshold=0.01 fdr_method=database_grounded + +# Compute-features overrides +winnow compute-features dataset_output_path=results/features.csv labelled=false ``` ### Nested parameters @@ -126,6 +130,37 @@ dataset_output_path: results/calibrated_dataset.csv - `model_output_dir`: Where to save trained model - `dataset_output_path`: Where to save calibrated training results +## Compute-features configuration + +### Main config (`configs/compute_features.yaml`) + +Loads data like `train.yaml` (same `defaults`: `residues`, `calibrator`, `data_loader`), runs `ProbabilityCalibrator.compute_features` only, and writes one CSV. No `model_output_dir` and no MLP training. + +```yaml +defaults: + - _self_ + - residues + - calibrator + - data_loader: instanovo + +dataset: + spectrum_path_or_directory: data/spectra.ipc + predictions_path: data/predictions.csv + +dataset_output_path: results/metadata.csv +filter_empty_predictions: true +labelled: true +``` + +**Key parameters:** + +- `dataset.*`, `data_loader`: Same meaning as in training config +- `dataset_output_path`: CSV path for metadata after feature computation +- `filter_empty_predictions`: If true, apply the same empty-prediction filter as train/predict +- `labelled`: If true, spectrum data must include `sequence` (ground truth); runs feature `prepare()` (needed for e.g. `RetentionTimeFeature`). If false, you must not include `retention_time_feature` in `calibrator.features` (validation error otherwise) + +The feature set is the `calibrator.features` block from `calibrator.yaml` (shared with training). Override or drop features with Hydra the same way as for `winnow train`. + ### Calibrator config (`configs/calibrator.yaml`) Controls model architecture and calibration features: @@ -577,6 +612,9 @@ winnow config train # View prediction configuration winnow config predict +# View compute-features configuration +winnow config compute-features + # View configuration with overrides winnow config train data_loader=mztab model_output_dir=custom/path winnow config predict fdr_method=database_grounded fdr_control.fdr_threshold=0.01 @@ -615,6 +653,7 @@ my_configs/ ├── residues.yaml # Override residue masses/modifications ├── calibrator.yaml # Override calibrator features ├── train.yaml # Override training config (if needed) +├── compute_features.yaml # Override compute-features config (if needed) ├── predict.yaml # Override prediction config (if needed) ├── data_loader/ # Override data loaders (if needed) │ └── instanovo.yaml diff --git a/tests/datasets/test_data_loaders.py b/tests/datasets/test_data_loaders.py index 4f43a1ef..a0bbdc4c 100644 --- a/tests/datasets/test_data_loaders.py +++ b/tests/datasets/test_data_loaders.py @@ -686,14 +686,14 @@ def test_evaluate_no_label_columns_when_no_labels(self, loader): dataset = pd.DataFrame({"prediction": [["A", "G"]]}) result = loader._evaluate_predictions(dataset, has_labels=False) assert "correct" not in result.columns - assert "valid_peptide" not in result.columns + assert "valid_sequence" not in result.columns assert "num_matches" not in result.columns - def test_evaluate_adds_valid_peptide_when_has_labels(self, loader): + def test_evaluate_adds_valid_sequence_when_has_labels(self, loader): dataset = pd.DataFrame({"sequence": [["A", "G"]], "prediction": [["A", "G"]]}) result = loader._evaluate_predictions(dataset, has_labels=True) - assert "valid_peptide" in result.columns - assert result["valid_peptide"].iloc[0] + assert "valid_sequence" in result.columns + assert result["valid_sequence"].iloc[0] def test_evaluate_correct_flag_true_on_full_match(self, loader): seq = ["P", "E", "P"] diff --git a/winnow/configs/compute_features.yaml b/winnow/configs/compute_features.yaml new file mode 100644 index 00000000..7ef851e6 --- /dev/null +++ b/winnow/configs/compute_features.yaml @@ -0,0 +1,17 @@ +# --- Compute calibration features (no MLP fit / no model save) --- +defaults: + - _self_ + - residues + - calibrator + - data_loader: instanovo # Options: instanovo, mztab, pointnovo, winnow + +dataset: + # Dataset paths (same as train): + spectrum_path_or_directory: examples/example_data/spectra.ipc + predictions_path: examples/example_data/predictions.csv + +dataset_output_path: results/metadata.csv +# Drop rows with empty or non-list predictions (same helper as train/predict). +filter_empty_predictions: true +# If true, compute features for a labelled dataset (needed for RetentionTimeFeature iRT mapping, where we train a model to predict iRT from retention time using a high-confidence labelled subset of the dataset). +labelled: true diff --git a/winnow/datasets/calibration_dataset.py b/winnow/datasets/calibration_dataset.py index 78193d06..4146f7c0 100644 --- a/winnow/datasets/calibration_dataset.py +++ b/winnow/datasets/calibration_dataset.py @@ -50,21 +50,10 @@ def save(self, data_dir: Path) -> None: data_dir (Path): Directory to save the dataset. This will contain `metadata.csv` and optionally, `predictions.pkl` for serialized beam search results. """ - data_dir.mkdir(parents=True, exist_ok=True) - with (data_dir / "metadata.csv").open(mode="w") as metadata_file: - output_metadata = self.metadata.copy(deep=True) - if "sequence" in output_metadata.columns: - output_metadata["sequence"] = output_metadata["sequence"].apply( - tokens_to_proforma - ) - output_metadata["prediction"] = output_metadata["prediction"].apply( - tokens_to_proforma - ) - output_metadata.to_csv(metadata_file, index=False) + self.save_metadata(data_dir / "metadata.csv") if self.predictions: - with (data_dir / "predictions.pkl").open(mode="wb") as predictions_file: - pickle.dump(self.predictions, predictions_file) + self.save_predictions(data_dir / "predictions.pkl") @property def confidence_column(self) -> str: @@ -156,6 +145,47 @@ def filter_entries( return CalibrationDataset(predictions=predictions, metadata=metadata) + def save_metadata(self, path: Union[Path, str]) -> None: + """Save the dataset metadata to a CSV file. + + Args: + path (Union[Path, str]): Path to the output CSV file. + """ + if isinstance(path, str): + path = Path(path) + path.parent.mkdir(parents=True, exist_ok=True) + + output_metadata = self.metadata.copy(deep=True) + + if "sequence" in output_metadata.columns: + output_metadata["sequence"] = output_metadata["sequence"].apply( + tokens_to_proforma + ) + if "prediction" in output_metadata.columns: + output_metadata["prediction"] = output_metadata["prediction"].apply( + tokens_to_proforma + ) + if "prediction_untokenised" in output_metadata.columns: + output_metadata = output_metadata.drop(columns=["prediction_untokenised"]) + if "valid_sequence" in output_metadata.columns: + output_metadata = output_metadata.drop(columns=["valid_sequence"]) + if "valid_prediction" in output_metadata.columns: + output_metadata = output_metadata.drop(columns=["valid_prediction"]) + output_metadata.to_csv(path, index=False) + + def save_predictions(self, path: Union[Path, str]) -> None: + """Save the dataset predictions to a pickle file. + + Args: + path (Union[Path, str]): Path to the output pickle file. + """ + if isinstance(path, str): + path = Path(path) + path.parent.mkdir(parents=True, exist_ok=True) + + with path.open(mode="wb") as predictions_file: + pickle.dump(self.predictions, predictions_file) + def to_csv(self, path: Union[Path, str]) -> None: """Saves the dataset metadata to a CSV file. diff --git a/winnow/datasets/data_loaders.py b/winnow/datasets/data_loaders.py index fd085db8..e5934d2e 100644 --- a/winnow/datasets/data_loaders.py +++ b/winnow/datasets/data_loaders.py @@ -397,7 +397,7 @@ def _evaluate_predictions( pd.DataFrame: The processed dataframe. """ if has_labels: - dataset["valid_peptide"] = dataset["sequence"].apply( + dataset["valid_sequence"] = dataset["sequence"].apply( lambda peptide: isinstance(peptide, list) ) dataset["valid_prediction"] = dataset["prediction"].apply( @@ -807,7 +807,7 @@ def _evaluate_predictions( .map_elements( lambda x: isinstance(x, pl.Series), return_dtype=pl.Boolean ) - .alias("valid_peptide"), + .alias("valid_sequence"), ] ) diff --git a/winnow/scripts/main.py b/winnow/scripts/main.py index b897465c..a1d8d7d6 100644 --- a/winnow/scripts/main.py +++ b/winnow/scripts/main.py @@ -11,7 +11,6 @@ import typer import logging from rich.logging import RichHandler -from pathlib import Path # Lazy imports for heavy dependencies - only imported when actually needed if TYPE_CHECKING: @@ -127,18 +126,20 @@ def separate_metadata_and_predictions( # Separate out metadata from prediction and FDR metrics preds_and_fdr_metrics_cols = [ - confidence_column, "prediction", + confidence_column, "psm_fdr", "psm_q_value", ] - if "sequence" in dataset_metadata.columns: - preds_and_fdr_metrics_cols.append("sequence") # NonParametricFDRControl adds psm_pep column if isinstance(fdr_control, NonParametricFDRControl): preds_and_fdr_metrics_cols.append("psm_pep") + if "sequence" in dataset_metadata.columns: + preds_and_fdr_metrics_cols.append("sequence") + preds_and_fdr_metrics_cols.append("num_matches") + preds_and_fdr_metrics_cols.append("correct") dataset_preds_and_fdr_metrics = dataset_metadata[ - preds_and_fdr_metrics_cols + ["spectrum_id"] + ["spectrum_id"] + preds_and_fdr_metrics_cols ] dataset_metadata = dataset_metadata.drop(columns=preds_and_fdr_metrics_cols) return dataset_metadata, dataset_preds_and_fdr_metrics @@ -215,11 +216,85 @@ def train_entry_point( # Save the training dataset results logger.info(f"Final dataset: {len(annotated_dataset)} spectra") logger.info(f"Saving training dataset results to {cfg.dataset_output_path}") - annotated_dataset.to_csv(cfg.dataset_output_path) + annotated_dataset.save_metadata(cfg.dataset_output_path) logger.info("Training pipeline completed successfully.") +def compute_features_entry_point( + overrides: Optional[List[str]] = None, + execute: bool = True, + config_dir: Optional[str] = None, +) -> None: + """Load a dataset, compute calibration features into metadata, and save CSV. + + Does not fit the MLP or write a calibrator checkpoint; reuses ``calibrator.features`` from Hydra like training. + + Args: + overrides: Optional list of config overrides. + execute: If False, only print the configuration and return. + config_dir: Optional path to custom config directory. + """ + from hydra import initialize_config_dir, compose + from hydra.utils import instantiate + from winnow.utils.config_path import get_primary_config_dir + + primary_config_dir = get_primary_config_dir(config_dir) + + with initialize_config_dir( + config_dir=str(primary_config_dir), + version_base="1.3", + job_name="winnow_compute_features", + ): + cfg = compose(config_name="compute_features", overrides=overrides) + + if not execute: + print_config(cfg) + return + + logger.info("Starting compute-features pipeline.") + logger.info(f"Compute-features configuration: {cfg}") + + labelled = bool(cfg.labelled) + if not labelled and cfg.calibrator.features.retention_time_feature is not None: + raise ValueError( + f"Compute-features config setting labelled={labelled}, but standalone feature computation for RetentionTimeFeature is not supported for unlabelled datasets.\n" + f"Please remove RetentionTimeFeature from the calibration feature set or use a labelled dataset with labelled=True." + ) + + logger.info("Loading dataset.") + data_loader = instantiate(cfg.data_loader) + + dataset_params = dict(cfg.dataset) + dataset_params["data_path"] = dataset_params.pop("spectrum_path_or_directory") + dataset_params["predictions_path"] = dataset_params.pop("predictions_path", None) + + dataset = data_loader.load(**dataset_params) + + logger.info(f"Loaded: {len(dataset.metadata)} spectra") + + if labelled and "sequence" not in dataset.metadata.columns: + raise ValueError( + "Labelled dataset must contain a 'sequence' column with ground-truth sequences." + ) + + if cfg.filter_empty_predictions: + logger.info("Filtering dataset for empty predictions.") + dataset = filter_dataset(dataset) + logger.info(f"After filtering: {len(dataset.metadata)} spectra") + + logger.info("Instantiating calibrator from config.") + calibrator = instantiate(cfg.calibrator) + + logger.info(f"Computing features with labelled={labelled}.") + calibrator.compute_features(dataset, labelled=labelled) + + logger.info(f"Saving dataset with features to {cfg.dataset_output_path}") + dataset.save_metadata(cfg.dataset_output_path) + + logger.info("Compute-features pipeline completed successfully.") + + def predict_entry_point( overrides: Optional[List[str]] = None, execute: bool = True, @@ -254,6 +329,7 @@ def predict_entry_point( return from winnow.calibration.calibrator import ProbabilityCalibrator + from winnow.datasets.calibration_dataset import CalibrationDataset from winnow.fdr.database_grounded import DatabaseGroundedFDRControl logger.info("Starting prediction pipeline.") @@ -312,11 +388,12 @@ def predict_entry_point( dataset_metadata, dataset_preds_and_fdr_metrics = separate_metadata_and_predictions( dataset_metadata, fdr_control, cfg.fdr_control.confidence_column ) - output_folder = Path(cfg.output_folder) - output_folder.mkdir(parents=True, exist_ok=True) - dataset_metadata.to_csv(output_folder.joinpath("metadata.csv")) - dataset_preds_and_fdr_metrics.to_csv( - output_folder.joinpath("preds_and_fdr_metrics.csv") + + CalibrationDataset(metadata=dataset_metadata).save_metadata( + cfg.output_folder + "/" + "metadata.csv" + ) + CalibrationDataset(metadata=dataset_preds_and_fdr_metrics).save_metadata( + cfg.output_folder + "/" + "preds_and_fdr_metrics.csv" ) logger.info("Prediction pipeline completed successfully.") @@ -362,6 +439,44 @@ def train( train_entry_point(overrides, config_dir=config_dir) +@app.command( + name="compute-features", + help=( + "Compute calibration features and write enriched metadata to CSV.\n\n" + "Loads the dataset using the same options as training, instantiates the calibrator " + "feature stack from config, runs feature computation (``prepare`` when labelled=true), " + "and saves the result without fitting the MLP or saving a model.\n\n" + "[bold cyan]Quick start:[/bold cyan]\n" + " [dim]winnow compute-features[/dim] # Uses config/compute_features.yaml\n\n" + "[bold cyan]Override parameters:[/bold cyan]\n" + " [dim]winnow compute-features dataset_output_path=results/my_features.csv[/dim]\n" + " [dim]winnow compute-features run_prepare=false[/dim] # Skip feature.prepare()\n" + " [dim]winnow compute-features filter_empty_predictions=false[/dim]\n\n" + "[bold cyan]Custom config directory:[/bold cyan]\n" + " [dim]winnow compute-features --config-dir /path/to/configs[/dim]\n" + " [dim]winnow compute-features -cp ./my_configs[/dim]\n\n" + "[bold cyan]Feature set:[/bold cyan]\n" + " Reuses [dim]calibrator.features[/dim] from calibrator.yaml; override with Hydra " + "(e.g. drop a feature with [dim]'~calibrator.features.retention_time_feature'[/dim])." + ), + context_settings={"allow_extra_args": True, "ignore_unknown_options": True}, +) +def compute_features( + ctx: typer.Context, + config_dir: Annotated[ + Optional[str], + typer.Option( + "--config-dir", + "-cp", + help="Path to custom config directory (relative or absolute). See documentation for advanced usage.", + ), + ] = None, +) -> None: + """Compute calibration features and save metadata CSV.""" + overrides = ctx.args if ctx.args else None + compute_features_entry_point(overrides, config_dir=config_dir) + + @app.command( name="predict", help=( @@ -435,6 +550,34 @@ def config_train( train_entry_point(overrides, execute=False, config_dir=config_dir) +@config_app.command( + name="compute-features", + help=( + "Display the resolved compute-features configuration without running the pipeline.\n\n" + "[bold cyan]Usage:[/bold cyan]\n" + " [dim]winnow config compute-features[/dim]\n" + " [dim]winnow config compute-features dataset_output_path=out.csv[/dim]\n" + " [dim]winnow config compute-features --config-dir /path/to/configs[/dim]\n" + " [dim]winnow config compute-features -cp ./my_configs[/dim]" + ), + context_settings={"allow_extra_args": True, "ignore_unknown_options": True}, +) +def config_compute_features( + ctx: typer.Context, + config_dir: Annotated[ + Optional[str], + typer.Option( + "--config-dir", + "-cp", + help="Path to custom config directory (relative or absolute). See documentation for advanced usage.", + ), + ] = None, +) -> None: + """Display the resolved compute-features configuration.""" + overrides = ctx.args if ctx.args else None + compute_features_entry_point(overrides, execute=False, config_dir=config_dir) + + @config_app.command( name="predict", help=(