Benchmarking peptide detectability predictors across mass spectrometry platforms and acquisition strategies

[xuxama]

aim of the new module

A large number of computational tools have been developed to predict peptide detectability in mass spectrometry experiments. However, whether a peptide is detected depends strongly on the instrument platform and particularly on the acquisition strategy (e.g., DDA vs DIA). As a result, predictors trained or optimized for one experimental setup may not generalize well to others.

This module aims to systematically benchmark peptide detectability predictors across different mass spectrometry platforms and acquisition types, allowing users to evaluate how well their predictor ranks peptides that are experimentally observed under specific experimental conditions.

full description of the new module

# Components

The module consists of two core components: (1) experimentally derived detectability ground truth and (2) a theoretical peptide universe used as input for prediction tools.

1. Detectability ground truth:

Ground truth detectability data will be derived from the LFQ benchmark datasets published by Van Puyvelde et al. These datasets include experiments performed with different species mixtures, instruments and acquisition strategies:

- DDA dataset: https://doi.org/10.1038/s41597-022-01216-6
- DIA dataset: https://doi.org/10.64898/2026.01.29.702266

Both studies analyze three samples with different ratios of human, yeast and E. coli proteins and were acquired on multiple MS platforms.

To establish the ground truth:
- DDA datasets will be analyzed using Sage
- DIA datasets will be analyzed using DIA-NN

Peptides identified with these workflows will be considered experimentally detected peptides. Because datasets span multiple platforms and acquisition types, this process will generate separate ground-truth detectability sets for each acquisition strategy and each instrument platform. This allows predictor performance to be evaluated under different experimental conditions.

2. Theoretical peptide universe

The benchmark will also provide a complete theoretical peptide universe, generated through an in silico digestion of the human, yeast and E. coli proteomes. This peptide list represents all peptides that could theoretically be observed in the benchmark samples. It serves as the common input set for all detectability predictors.

Users can download this peptide list and run their predictor to generate a ranked list of peptides ordered by predicted detectability.

# User workflow

1. Download the provided theoretical peptide universe
2. Run a detectability predictor on this peptide list
3. Produce ranked output file in standard format (e.g. CSV with columns 'peptide' and 'score')
4. Upload file to ProteoBench

The platform will automatically compare the ranking with the ****precomputed ground truth datasets and calculate the benchmark metrics.

Because the ground truth differs for each instrument and acquisition strategy, the uploaded predictions will be evaluated against all relevant datasets automatically. Users therefore only need to upload a single ranked peptide list, and they can evaluate their predictors performance for different settings.

# Metrics

The module evaluates how strongly experimentally observed peptides are prioritized in the predicted ranking.

1. Normalized enrichment score

This metric quantifies how strongly detected peptides are enriched at the top of the ranked list.

N = total number of peptides in the ranked list
K = number of experimentally detected peptides
r₁, r₂, …, r_K = ranks of detected peptides

Average rank of detected peptides:

m = (r₁ + r₂ + … + r_K) / 2

Best possible mean rank of observed peptides (theoretical upper bound):

m_best = (K+1)/2

Worst possible mean rank of observed peptides (theoretical lower bound):

m_worst = m_worst = (2N - K + 1) / 2

Normalized enrichment score:

E = (m_worst - m) / (m_worst - m_best)

	E == 1: perfect ranking (all detected peptides at top)
	E == 0.5: random ranking
	E == 0: reversed ranking (all detected peptides at bottom)


2. Top-k precision

This metric measures how many experimentally detected peptides appear among the highest-ranked predictions.

precision@k = detected peptides within top k / k

# Predictors

Potential published predictors that could be included are DeepDetect, PFly, DeepMSPeptide and DbyDeep.

# Visualization

Benchmark results will be visualized using a scatter plot, where each point represents a detectability predictor.

- x-axis: top-k recall
- y-axis: normalized enrichment score

Different acquisition strategies (DDA, DIA) will be organized into separate tabs. Within each tab, users can select the instrument platform to view the corresponding results.

potential external expert(s)

No response

Will you be able to work on the implementation (coding) yourself, with additional help from the ProteoBench maintainers?

• yes
• no

any other information

In future versions of the module, the benchmark may be extended to include targeted acquisition strategies. At present, however, no publicly available benchmark dataset exists with comparable sample compositions for targeted workflows. Therefore, targeted acquisition will not be included in the initial release of this module.

[mlocardpaulet]

Hi,

this is a great idea.
To determine the set of "experimentally detected peptides", you plan to use SAGE for DDA and DIA-NN for DIA. I wonder why you would not combine several search outputs from the ProteoBench modules, with different tool outputs? This could reduce biases comming from the search engines.
With the same thinking process, for the peptide universe: maybe not all proteins of the HYE proteomes are present in the samples. Maybe you could consider using as peptide universe all the tryptic peptides that would theoretically result from the digestion of all the proteins identified in the ProteoBench search outputs?

I am not super familiar with peptide detectability predictors, only used a bit peptideRanger. How are modifications considered? Are they?

[mlocardpaulet]

Very naive question: to evaluate peptide detectability, one limitation of the data is that the initial amount of peptides is not equal for all proteins (they don't have the same quantity in the proteomes). Would this be an issue? Would there be a way to control for it?

[xuxama]

Using multiple search outputs sounds good. We could take the union, intersection or maybe all peptides that were detected by at least two engines. And I also agree with your suggestion about the peptide universe.

I am quite sure most predictors do not support modifications. They just report detectability for the stripped peptide sequence. I think we should predict detectability for the stripped peptides and consider them detected if any form (modified or unmodified) is detected. Maybe we could make including PTMs optional for predictors that do support them.

Its true that detectability is influenced by abundance, especially in DDA. In this case, however, I think that is an effect we would like to capture rather than remove. Since most predictors to my knowledge are trained on DDA data, they may implicitly learn this bias. Doing the benchmark on both DDA and DIA can reveal if the predictors predict intrinsic detectability (regardless of abundance) or effective detectability under specific MS acquisition conditions. However, since the datasets include different mixture ratios, we could maybe use this somehow to assess how robust predictors are to changes in abundance.

[rodvrees]

One more thing is that we have to be careful with train-test leakage. It’s very likely that peptides from a human, e.coli and yeast mixture will be present in the training data of the models. I really like @mlocardpaulet’s idea of using the existing data, but this is something to take into account.
We could ask users to upload their training data to calculate overlap, and report this.

[mlocardpaulet]

Have you seen this paper? https://spj.science.org/doi/10.34133/csbj.0037

[veitveit]

I see some potential issues in this approach:

• Higher or lower detectability estimation needs to be independent of the protein abundance. Therefore, the benchmarking should only happen within each protein
• Database search might be a factor as it also impacts detectability
• Given that the data has rather common species, the models will most likely have seen these peptides
• Ignoring PTMs is important due to the prediction models, but also will introduce a bias

[xuxama]

Updates following in-person meeting

Here is a summary of the changes we agreed on during our meeting:

Ground truth dataset

• We will not use the Van Puyvelde, Devreese et al. LFQ benchmark datasets. Because those datasets contain human, yeast and E. coli peptides, there is a high risk of train-test leakage with most predictors. Instead, we will look for high-quality datasets from unusual or less common species that are unlikely to appear in predictor training sets. FASTA annotation quality should be verified for any candidate species.
• For the ground truth definition, we will evaluate both the union and intersection of peptides detected across all search engines and test which works best.
• Since many tools require proteins as input and not peptide, we will offer the choice between downloading the in silico digested peptide universe or the protein fasta containing all proteins detected in the ground truth.
• Proteins with only a single identified peptide will be removed from the downloadable FASTA and in silico digested peptide universe.
• Since most (if not all) current predictors do not support PTMs, we will only evaluate unmodified peptides. A peptide in the theoretical universe will only be labeled as "observed" if it was detected in its unmodified form. If tools supporting PTMs emerge in the future, we can extend the module at that point.
• Peptides shared across multiple proteins will be excluded from the theoretical universe and from evaluation. This is necessary because some predictors use protein sequence context when scoring peptides, meaning the same peptide can receive a different score depending on which protein it is assigned to. However, search engine output does not record which protein a shared peptide originated from — it only tells us whether the peptide was detected. This creates an unfair evaluation situation: a predictor might correctly rank a shared peptide highly in one protein context but low in another, and if the low-ranked instance is matched against an "observed" label, the tool gets penalized despite having made a correct prediction elsewhere. Removing shared peptides entirely avoids this ambiguity. We will need to monitor how many peptides this removes and assess whether sufficient peptides remain for a robust benchmark.

Evaluation

• Metrics will be calculated per-protein rather than across a single exhaustive peptide list. Protein abundance strongly influences whether a peptide is detected, so a global ranking could unfairly penalize predictors for correctly ranking peptides from low-abundance proteins that happen not to be observed.
• Additional in-depth plots:
  • Precision–recall plot (per protein): useful for tools run with e.g. 0 missed cleavages, which may have high precision but lower recall.
  • Normalized rank enrichment plot
  • Number of search engines vs. score plot: varies ground truth stringency from peptides detected in one search engine up to those detected in all, showing how predictor performance changes with ground truth confidence.

User workflow

• Users will be asked to provide their training peptides so we can calculate overlap with the benchmark ground truth. Overlap will be visualized by encoding point density in the scatter plot, so users can interpret results with appropriate caution.
• Tools that take proteins as input often require parameters like missed cleavages and peptide length ranges. We will recommend users apply the same parameters as the peptide universe (e.g. suggested: 2 missed cleavages, length 7–30). We will check submitted parameters and report how much the user's theoretical peptide list deviates from ours.
• Users will be asked to provide a parameter/config file or command line arguments. For established tools, ProteoBench will include parsers for known formats. For new tools, we will define a standard format and encourage authors to reach out so we can add support.

[xuxama]

Updates following meeting with Matthew The

We had a discussion with Matthew The to get his perspective on the ground truth design.

Ground truth construction

Addressing training data overlap:

• to address the issue of training data overlap, we should include at least one sample type or organism unlikely to appear in any predictor's training data, and training data of each predictor should be documented in proteobench.
• given expected detectability differences between sample types, benchmarking only on uncommon species would make the module less useful, as it would not show results for sample types that are actually being used in the community

Including relevant sample types for the community:

• Dataset of >100 donors and ~30 tissue types (https://www.cell.com/cell/fulltext/S0092-8674(20)31078-3) as a candidate for tissue-specific ground truths (liver, brain, etc.).
• HeLa is a practical starting point given the large amount of publicly available data.
• ProteomicsDB has a "proteotypicity" metric, but it is not stratified by experimental setup and has currently limited Astral coverage.
• Subcellular fractionation was mentioned as a way to assess ranking consistency across protein abundance levels, but is not a suitable ground truth candidate due to its highly specific sample prep.

Including both common and uncommon sample types serves two purposes:

• Uncommon samples provide a fair comparison between predictors, as training data overlap is unlikely for any of them.
• Common samples allow users to identify the predictor that works best for their specific experimental setup, even if that predictor fails to generalise to other sample types.

Key open question

Before selecting datasets, we should empirically test whether per-protein peptide rankings differ across sample types and are stable between replicates, using the Jiang et al. dataset. Dataset selection should be delayed until this analysis is done.