Merge pull request #1065 from Proteobench/DIA_entrapment_peptide_level

Dia entrapment peptide level

Author: RobbinBouwmeester

.gitignore

@@ -153,9 +153,13 @@ jupyter_notebooks/analysis/
 testfolder
 .claude
 uv.lock
+test/mock_entrapment_datapoints
 
 # Large files
 *.gz
+jupyter_notebooks/
+testfolder/
+proteobench/score/static_files
 # Atomic-write / editor temp files
 *.tmp
 *.tmp.*

docs/available-modules/active-modules/13-entrapment-ion-dia-astral.md

@@ -0,0 +1,120 @@
+# DIA Ion Entrapment - Astral
+
+This module uses entrapment peptides to assess whether the false discovery rate (FDR) reported by a DIA search engine is reliable. It is based on the entrapment approach described in [Wen et al., 2025](https://www.nature.com/articles/s41592-025-02719-x).
+
+> **This module is in alpha stage. Results and interfaces may change.**
+
+## Purpose
+
+FDR control is a central assumption in proteomics data analysis. When a search engine reports 1% FDR, the actual proportion of false discoveries should not exceed 1%. This module tests that assumption empirically by spiking a known set of "entrapment" peptides into the search space. These peptides cannot be present in the sample (they are derived from a shuffled or decoupled database), so any identification of an entrapment peptide represents a false discovery.
+
+It is important to note that different search engines calculate the FDR on different levels - within a run, globally, on the precursor level, peptidoform level, protein group level, ... .
+This module tests the **global precursor ion level FDR**. Search engines that do not calculate the FDR on this level can not be benchmarked with this module.
+
+Three metrics are computed:
+
+- **Lower bound FDP**: minimum estimate of the false discovery proportion, based on the raw count of entrapment identifications.
+- **Combined FDP** (upper bound): corrected upper bound accounting for the 1:1 target-to-entrapment ratio in the search database.
+- **Paired FDP** (upper bound): refined upper bound using a paired peptide mapping, which accounts for entrapment peptides that are harder to identify than their target counterparts.
+
+Each workflow submission is classified as:
+
+- **Valid**: the upper bound FDP is lower than the reported FDR threshold.
+- **Inconclusive**: the lower bound is lower than the declared FDR but the upper bound is higher. This means that the actual FDP could be lower or higher than the reported FDR, therefore we can not say if the FDR is valid or not.
+- **Invalid**: even the lower bound is higher than the declared FDR threshold.
+
+## Critical requirements before running
+
+> **Do not rename the raw files after download.** File names are used for run mapping and must match exactly.
+
+> **Use the pre-digested entrapment FASTA — do not enable in-silico digestion.** The entrapment FASTA already contains peptide sequences (not full proteins). Enabling digestion in your search engine will lead to identified peptides without matched entrapments, which makes FDP calculation less accurate.
+
+> **Do not add any variable modifications.**
+
+## Data set
+
+The benchmark dataset consists of three technical replicates of a human plasma digest acquired on an Orbitrap Astral (Thermo Fisher Scientific) in DIA mode with a 15-minute gradient.
+
+Download the raw files from the ProteoBench server:
+
+- `LFQ_Astral_DIA_15min_50ng_Human_01`
+- `LFQ_Astral_DIA_15min_50ng_Human_02`
+- `LFQ_Astral_DIA_15min_50ng_Human_03`
+
+**It is imperative not to rename the files once downloaded.**
+
+The entrapment FASTA (`ProteoBenchFASTA_Entrapment_Human_with_contaminants_entrapment_pep.txt`) is available from the ProteoBench server. It contains human peptide sequences alongside a matched set of entrapment peptide sequences (suffixed `_p_target` to allow classification). **This file is pre-digested: do not apply enzymatic digestion in your search engine settings.**
+
+## Metric calculation
+
+ProteoBench reads the search engine output, maps runs to samples, and classifies each precursor identification as either a **target** or an **entrapment** hit based on the tag in the fasta.
+
+The three FDP estimates are computed from the resulting set and compared to the reported FDR threshold (inferred from the output file).
+
+## How to use
+
+### Input data for private visualisation
+
+The module currently accepts DIA-NN output (`report.tsv` or `report.parquet`). Use the suggested parameters in Table 1 for a fair comparison between tools.
+
+**Table 1. Suggested parameters**
+
+| Parameter | Value |
+|---|---|
+| PSM / precursor FDR | 0.01 |
+| Spectral library | Predicted from entrapment FASTA |
+| Digestion | **None** (FASTA is pre-digested) |
+| Fixed modifications | Carbamidomethylation (C) |
+| Variable modifications | **None** |
+| Precursor charge range | 1–5 |
+| Precursor m/z range | 400–1000 |
+| Fragment m/z range | 100–1800 |
+
+### Submit your run for public usage
+
+After uploading and privately inspecting your results, you can submit the run for public display. Upload the parameter file associated with your search (see tool-specific sections below), fill in the `Comments for submission` field if needed, confirm the metadata is correct, and press `I really want to upload it`.
+
+You will receive a link to a GitHub pull request. Save it — it contains your run's unique identifier and allows you to communicate with the ProteoBench maintainers. Accepted submissions become publicly visible within a few working days.
+
+**Table 2. Input files required for metric calculation and public submission**
+
+| Tool | Input file | Parsed FDR Column | Parameter file |
+|---|---|---|---|
+| DIA-NN | `report.tsv` or `report.parquet` | Lib.Q.Value | `report.log.txt` |
+
+## Tool-specific settings
+
+### [DIA-NN](https://github.com/vdemichev/DiaNN)
+
+1. Import the raw `.raw` files.
+2. Add the entrapment FASTA. Do not enable "Contaminants" — contaminants are already included in the FASTA.
+3. **Disable in-silico digestion.** The FASTA is pre-digested; use '--cut ' in the additional parameter fields to disable enzymatic cleavage.
+4. Enable library-free search / FASTA-based library generation (activates deep-learning prediction of spectra, RTs, and IMs).
+5. Do not set verbosity / log level higher than 1, otherwise parameter parsing will fail.
+6. Upload `report.tsv` or `report.parquet` for metric calculation, and `report.log.txt` for public submission.
+
+## Result description
+
+After uploading, you will see the FDP bounds plotted against the FDR estimates by the search engine. A valid (conservative) FDR calculation shows as both the upper and lower bound being higher than the FDR estimation.
+
+You can also compare the results with other submissions in the 'View Public + New Results' Tab. The following plots are shown:
+
+- **Forest plot**: one horizontal interval per workflow, showing the lower bound FDP to the paired FDP upper bound. The declared FDR threshold is marked with a diamond.
+- **FDP / FDR ratio vs # IDs**: scatter plot of paired FDP divided by the declared FDR (x-axis) against the number of identifications at 1% FDR (y-axis). Points to the left of x = 1 have an empirical FDP below the declared threshold.
+
+## Define parameters
+
+For public submission, the following parameters are extracted from the parameter file. If a parameter is not in the file, add it in `Comments for submission`.
+
+- Software tool name and version
+- FDR threshold for PSM / precursor, peptide, and protein level
+- Precursor and fragment m/z range
+- Precursor mass tolerance
+- Fragment mass tolerance
+- Fixed and variable modifications
+- Minimum and maximum precursor charge
+- Enzyme (should be "None" for this module)
+
+**DISCLAIMER**: Parameter files may contain file paths that reveal personal usernames, system architecture, or directory structures (FASTA location, raw data location, tool installation paths). Review and sanitize file paths before submission to avoid disclosing institutional or personal identifiers.
+
+Once submitted, a pull request link is shown. Save it to track your submission. Contact us via a [GitHub issue](https://github.com/Proteobench/ProteoBench/issues/new) or [email](mailto:proteobench@eubic-ms.org?subject=ProteoBench_query) if you encounter any problems.

docs/module_grid_generated.rst

@@ -81,6 +81,16 @@ Available ProteoBench modules
         +++
         `Web app <https://proteobench.cubimed.rub.de/denovo_DDA_HCD>`__ · `Documentation <https://proteobench.readthedocs.io/en/latest/available-modules/active-modules/11-denovo-dda-hcd/>`__
 
+    .. grid-item-card:: Entrapment DIA ion Astral
+        :columns: 6 6 4 4
+        :padding: 1
+
+        :bdg-info:`in development`
+        ^^^
+        
+        +++
+        `Web app <https://proteobench.cubimed.rub.de/Entrapment_DIA_ion_Astral>`__ · `Documentation <https://proteobench.readthedocs.io/en/latest/available-modules/active-modules/13-entrapment-ion-dia-astral/>`__
+
     .. grid-item-card:: Quant LFQ DIA ion Plasma
         :columns: 6 6 4 4
         :padding: 1

proteobench/datapoint/entrapment_datapoint.py

@@ -0,0 +1,180 @@
+"""
+This module provides functionality for handling and processing quantitative datapoints in the ProteoBench framework.
+"""
+
+from __future__ import annotations
+
+import dataclasses
+import hashlib
+import logging
+from collections import ChainMap, defaultdict
+from dataclasses import dataclass
+from datetime import datetime
+from typing import Any, Dict
+
+import numpy as np
+import pandas as pd
+
+import proteobench
+from proteobench.datapoint.datapoint_base import DatapointBase
+from proteobench.score.entrapmentscores import EntrapmentScores
+
+
+@dataclass
+class EntrapmentDatapoint(DatapointBase):
+    """
+    A data structure used to store the results of a entrapment benchmark run.
+
+    This class extends DatapointBase to implement entrapment-specific metrics and metadata
+    storage for LFQ benchmarking runs.
+
+    Attributes:
+        id (str): Unique identifier for the benchmark run.
+        software_name (str): Name of the software used in the benchmark.
+        software_version (str): Version of the software.
+        search_engine (str): Name of the search engine used.
+        search_engine_version (str): Version of the search engine.
+        ident_fdr_psm (float): False discovery rate for PSMs.
+        ident_fdr_peptide (float): False discovery rate for peptides.
+        ident_fdr_protein (float): False discovery rate for proteins.
+        enable_match_between_runs (bool): Whether matching between runs is enabled.
+        precursor_mass_tolerance (str): Mass tolerance for precursor ions.
+        fragment_mass_tolerance (str): Mass tolerance for fragment ions.
+        enzyme (str): Enzyme used for digestion.
+        allowed_miscleavages (int): Number of allowed miscleavages.
+        min_peptide_length (int): Minimum peptide length.
+        max_peptide_length (int): Maximum peptide length.
+        is_temporary (bool): Whether the data is temporary.
+        intermediate_hash (str): Hash of the intermediate result.
+        results (dict): A dictionary of metrics for the benchmark run.
+        nr_id_features (int): Number of identified features.
+        lower_bound_FDP (float): estimated false discovery proportion based on entrapment IDs.
+        combined_FDP (float): estimated False discovery proportion based on entrapment IDs.
+        paired_FDP (float): estimated False discovery proportion based on entrapment IDs.
+        reported_fdr_parsed_from_input (float): FDR threshold inferred from the input data (max Q-value).
+        comments (str): Any additional comments.
+        proteobench_version (str): Version of the Proteobench tool used.
+    """
+
+    id: str = None
+    software_name: str = None
+    software_version: int = 0
+    search_engine: str = None
+    search_engine_version: int = 0
+    ident_fdr_psm: int = 0
+    ident_fdr_peptide: int = 0
+    ident_fdr_protein: int = 0
+    enable_match_between_runs: bool = False
+    precursor_mass_tolerance: str = None
+    fragment_mass_tolerance: str = None
+    enzyme: str = None
+    allowed_miscleavages: int = 0
+    min_peptide_length: int = 0
+    max_peptide_length: int = 0
+    is_temporary: bool = True
+    intermediate_hash: str = ""
+    results: dict = None
+    nr_id_features: int = 0
+    lower_bound_FDP: float = np.nan
+    combined_FDP: float = np.nan
+    category_combined: str = ""
+    category_paired: str = ""
+    paired_FDP: float = np.nan
+    reported_fdr_parsed_from_input: float = np.nan
+    fdp_curve: dict = None
+    comments: str = ""
+    proteobench_version: str = ""
+
+    def generate_id(self) -> None:
+        """
+        Generate a unique ID for the benchmark run by combining the software name and a timestamp.
+
+        This ID is used to uniquely identify each run of the benchmark.
+        """
+        time_stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+        self.id = "_".join([self.software_name, str(time_stamp)])
+        logging.info(f"Assigned the following ID to this run: {self.id}")
+
+    @staticmethod
+    def generate_datapoint(
+        intermediate: pd.DataFrame,
+        input_format: str,
+        user_input: dict,
+    ) -> pd.Series:
+        """
+        Generate a Datapoint object containing metadata and results from the benchmark run.
+
+        Parameters
+        ----------
+        intermediate : pd.DataFrame
+            The intermediate DataFrame containing benchmark results.
+        input_format : str
+            The format of the input data (e.g., file format).
+        user_input : dict
+            User-defined input values for the benchmark.
+        default_cutoff_min_prec : int, optional
+            The default minimum precursor cutoff value. Defaults to 3.
+        max_nr_observed : int, optional
+            Maximum nr_observed value to calculate metrics for. If None, defaults to 6.
+
+        Returns
+        -------
+        pd.Series
+            A Pandas Series containing the Datapoint's attributes as key-value pairs.
+        """
+        current_datetime = datetime.now()
+        formatted_datetime = current_datetime.strftime("%Y%m%d_%H%M%S_%f")
+
+        if "comments_for_plotting" not in user_input.keys():
+            user_input["comments_for_plotting"] = ""
+
+        try:
+            user_input = defaultdict(
+                user_input.default_factory,  # Preserve the default factory
+                {key: ("" if value is None else value) for key, value in user_input.items()},
+            )
+        except AttributeError:
+            user_input = {key: ("" if value is None else value) for key, value in user_input.items()}
+
+        result_datapoint = EntrapmentDatapoint(
+            id=input_format + "_" + user_input["software_version"] + "_" + formatted_datetime,
+            software_name=input_format,
+            software_version=user_input["software_version"],
+            search_engine=user_input["search_engine"],
+            search_engine_version=user_input["search_engine_version"],
+            ident_fdr_psm=user_input["ident_fdr_psm"],
+            ident_fdr_peptide=user_input["ident_fdr_peptide"],
+            ident_fdr_protein=user_input["ident_fdr_protein"],
+            enable_match_between_runs=user_input["enable_match_between_runs"],
+            precursor_mass_tolerance=user_input["precursor_mass_tolerance"],
+            fragment_mass_tolerance=user_input["fragment_mass_tolerance"],
+            enzyme=user_input["enzyme"],
+            allowed_miscleavages=user_input["allowed_miscleavages"],
+            min_peptide_length=user_input["min_peptide_length"],
+            max_peptide_length=user_input["max_peptide_length"],
+            intermediate_hash=str(hashlib.sha1(intermediate.to_string().encode("utf-8")).hexdigest()),
+            comments=user_input["comments_for_plotting"],
+            proteobench_version=proteobench.__version__,
+        )
+
+        result_datapoint.generate_id()
+        metrics = EntrapmentDatapoint.get_metrics(intermediate)
+
+        result_datapoint.reported_fdr_parsed_from_input = metrics["reported_fdr_parsed_from_input"]
+        result_datapoint.nr_id_features = metrics["nr_id_features"]
+        result_datapoint.lower_bound_FDP = metrics["lower_bound_FDP"]
+        result_datapoint.combined_FDP = metrics["combined_FDP"]
+        result_datapoint.paired_FDP = metrics["paired_FDP"]
+        result_datapoint.category_combined = metrics["category_combined"]
+        result_datapoint.category_paired = metrics["category_paired"]
+        result_datapoint.fdp_curve = metrics["fdp_curve"]
+
+        result_datapoint.results = metrics
+        results_series = pd.Series(dataclasses.asdict(result_datapoint))
+
+        return results_series
+
+    @staticmethod
+    def get_metrics(intermediate: pd.DataFrame) -> Dict[str, Any]:
+        metrics = EntrapmentScores.calculate_metrics(intermediate)
+        return metrics

proteobench/exceptions.py

@@ -62,6 +62,12 @@ def __init__(self, message):
         super().__init__(self.message)
 
 
+class EntrapmentError(ProteobenchError):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
 class PlotError(ProteobenchError):
     def __init__(self, message):
         self.message = message

proteobench/io/params/diann.py

@@ -485,8 +485,10 @@ def extract_params(
                 parameters[proteobench_setting] = parse_setting(proteobench_setting, cmdline_dict[cmd_setting])
 
     # Parse cut parameter to standard enzyme name
-    if "enzyme" not in parameters.keys():  # This happens when running fragpipe-diann or if kept as default in the GUI
+    if "enzyme" not in parameters.keys():  # This happens when running fragpipe-diann
         parameters["enzyme"] = "Trypsin/P"
+    elif parameters["enzyme"] is True:  # bare --cut flag with no value = no-digestion mode
+        parameters["enzyme"] = "No digestion"
     elif parameters["enzyme"] == "K*,R*":
         parameters["enzyme"] = "Trypsin/P"
     elif parameters["enzyme"] == "K*,R*,!P":