DDA and DIA quantification - protein groups

[mlocardpaulet]

This module will be the same as DDA quantification - precursor ions, but for protein group quantities. For this, the input will be benchmark run outputs with protein group quantities, e.g. MaxQuant proteinGroups.txt

[mburq]

Hello,
I'm wondering whether there's been any work on this module yet.
I'm currently working on a protein group quantification algorithm, and currently testing on the dataset from the DDA quant module (precursor Ion, Q-Exactive).
I'm thinking the metrics would be the same: #protein groups identified on y axis and [median|mean] absolute difference between measured and expected log_2 transformed fold change.

If that's of interest, I can try to submit a PR for a protein-group quantification module based on these same files.

[julianu]

Hej,
I would argue the big problem is: what is good and (in tendency) true. Just having high numbers of protein groups is obviously not what makes the workflow well performing. For peptides / ions this might also not be the best, but there we don't run into the protein inference problem.
Just my immediate thoughts, but I think there was some discussion at the beginning of the project as well?

[mburq]

If you have a record of that discussion I would be interested in learning what options were considered at the time.

My (possibly naïve) approach would be something along those lines:

• Only include canonical protein sequences in the initial Search FASTA, to reduce the number of shared peptides
• Only consider "singleton" protein groups, and discard from the analysis any groups that contain more than one protein

One downside of this analysis is that it is not useful to measure improvements to the protein inference problem.
It would however be a helpful way to measure the quality of the ion_quant --> protein_quant rollup algorithm.

The obvious FDR tradeoff between quantity and quality remains the same as the other existing benchmarks for peptides/ions.

[mburq]

Hello, quick follow-up on the thread above, are there thoughts on moving forward with this?

[mlocardpaulet]

Hi,

Yes, one big issue is to match protein groups. I mean...
Depending on the search engine/pipeline, and the parameters, you can have one accession in several protein groups. And parsimony can be applied or not (you can find the same accession in several protein groups), the order of the accessions in the groups may be different, etc...

It would definitely be great to start working on such module (we have not yet). @mburq, would you be interested to work on this? Would you be alone?
Maybe we could organise a meeting to discuss this?

[mburq]

Yes I'd be interested in working on this. I can take the lead / start doing this alone, but also happy to learn from your collective experience and collaborate with anyone else interested.
Happy to set up a meeting, I'm pretty available over the next 1-2 weeks.

[mlocardpaulet]

Perfect! Could you provide your email address so that we can arrange something?

[mburq]

Just reached out over email, let me know if you didn't receive it.

[julianu]

Sorry for the delay,
I didn't want to be to pessimistic with my post above and definitely would be interested. If someone takes the lead, I am happy to join (at least the discussion).

[mlocardpaulet]

Hi, we received a few requests for this module (using the mixes-species data from the current modules https://proteobench.cubimed.rub.de/Quant_LFQ_DDA_ion_QExactive and similar).
Could we start using the DIA Astral data? https://proteobench.readthedocs.io/en/stable/available-modules/active-modules/7-quant-lfq-ion-dia-Astral_2Th/

One suggestion is to use the following metric for the main plot:

• horizontal axis = mean/median epsilon after exclusion of protein groups containing different species and/or contaminant protein(s)
• vertical axis is a bit more complicated because some software tools do not apply the parsimony principle, and high numbers could be due to accessions being in several protein groups. So we could:
  - use a totally different metric (mean/median CVs for example)
  - report the total number of protein groups, with somewhere a metric that will give an idea of accession redundancy (proportion of groups with shared accessions?

[mlocardpaulet]

see PR #960

[mlocardpaulet]

One pb. that I encounter: in DIA-NN, gene names are required in the fasta file for some of the inference/summarization algorithms. And the gene names are not in the fasta that we provided for the current mixed-species modules.
Ideally, we would add the gene names in the fasta headers that we currently work with (adding the "Cont-" flag for the gene names in the contaminant proteins). We could then replace the fasta without gene names everywhere. This should not change results for the precursors and peptidoform-level modules, while allowing the use of gene-related parameters for protein-group-level modules.

[mlocardpaulet]

I made an annotated fasta (with full Uniprot fasta headers) from the ProteoBenchFASTA_MixedSpecies_HYE.fasta provided with the ProteoBench modules. Some accessions are already a bit old, so they do not have gene names, but this looks like something that could happen in real life, so it is not so bad to have it. I hope that using this fasta will lead to exactly the same results as using the original ProteoBenchFASTA_MixedSpecies_HYE.fasta in the precursor-/peptidoform-level modules.
@wolski I know that you asked for such an annotated fasta for a while, right? Could you please have a look, check that it looks fine, and test whatever you feel like testing with it? Let me know if you see something fishy.
In the meantime, I will use it for testing the protein-group-level module.

ProteoBenchFASTA_MixedSpecies_HYE_wGenes.fasta.zip

[mlocardpaulet]

From discussions during our in-person meeting:

1- A problem with the data we use (three-species mix): the peptides from the same species all have the same expected ratio. So issues with summarization won't lead to bigger error, or at least less than in more realistic samples.
2- We throw away the mixed-species groups because the accuracy that we calculate is based on the expected ratio (and which one would we pick?). Could we use the precision instead of the accuracy?
3- Split the metrics to have error calculated on the protein groups with only one accession, or with more than one.
4- We could only consider the protein accessions that have no shared peptide in theory (according to the fasta file). And we only report the accuracy error for this? With the number of accessions on the y-axis.

We go for 4.

Open question:
If we agree that the data set is not suited for protein-group-level benchmark (point 1), what sample could we use?

• pb that if we have protein groups, the ratio of which accession do we use as expected ratio for the benchmark?
• we could have a sequenced sample (with known set of isoforms) as ground truth. Maybe 2 cell lines that would be sequenced and that we could mix together in different proportions.