Skip to content

metrics.md

Latest commit

 

History

History
127 lines (97 loc) · 5.71 KB

metrics.md

File metadata and controls

127 lines (97 loc) · 5.71 KB
Raw

Metrics

run_metrics writes per-sequence metrics to decoded_sequences_metrics.csv. Metrics are grouped into sequence, optional structure, and optional immunogenicity columns.

Sequence Metrics

  • sequence_recovery_vs_wildtype: strict index-by-index identity versus the wildtype/input sequence. Higher is better.
  • sequence_recovery_vs_mpnn_reference: identity versus the LigandMPNN reference sequence. Higher is better.
  • sequence_recovery_vs_mpnn_argmax: identity versus the position-wise LigandMPNN argmax sequence. Higher is better.
  • highconf_recovery_vs_mpnn_reference_p90: recovery over high-confidence LigandMPNN positions. Higher is better.
  • mean_logprob: mean LigandMPNN log probability for the sequence. Higher is better.
  • perplexity: exponentiated negative mean log probability. Lower is better.
  • delta_mean_logprob_vs_mpnn_reference: change in mean log probability relative to the LigandMPNN reference.
  • delta_mean_logprob_vs_mpnn_argmax: change in mean log probability relative to the LigandMPNN argmax.

Global alignment metrics compare decoded sequences to wildtype:

  • needle_score_vs_wildtype: BLOSUM62-style global alignment score. Higher is better.
  • needle_score_norm_vs_wildtype: score normalized by the wildtype self-alignment score. Higher is better.
  • needle_identity_vs_wildtype: fraction of aligned columns with exact identity. Higher is better.
  • needle_similarity_vs_wildtype: fraction of aligned columns with exact or conservative substitutions. Higher is better.
  • needle_gap_fraction_vs_wildtype: fraction of aligned columns containing a gap. Lower is better.

Composite Ranking

sequence_composite_v1 is the default sequence ranking metric. It combines:

  • decode support from LigandMPNN probability/perplexity signals
  • recovery versus LigandMPNN reference and argmax sequences
  • wildtype identity/similarity
  • high-confidence/key residue recovery
  • sequence complexity penalties
  • relative length penalties

Structure Metrics

Structure metrics are disabled by default because they require an installed structure-prediction backend and are usually GPU-bound.

metrics:
  structure:
    enabled: true
    compute_tm_score: true
    model: protenix

Supported structure-prediction backends are Protenix and Boltz2.

Common structure columns:

  • structure_rmsd: global C-alpha RMSD after sequence-aware chain matching and superposition. Lower is better.
  • structure_tm_score: TM-align score against the reference structure. Higher is better.
  • structure_pLDDT: mean predicted local confidence. Higher is better.
  • structure_pae_mean: mean predicted aligned error when available. Lower is better.
  • structure_pae_local: PAE restricted to configured fixed/key residues when available. Lower is better.
  • structure_ptm: predicted TM-style global confidence when available. Higher is better.
  • structure_contact_recovery: fraction of reference residue contacts recovered. Higher is better.
  • structure_contact_map_similarity: contact-map Jaccard-style similarity. Higher is better.
  • structure_active_site_rmsd: RMSD over target_protein.activity.fixed_residues. Lower is better.
  • structure_binding_pocket_rmsd: pocket/key-residue RMSD. Lower is better.
  • structure_interface_rmsd: interface-residue RMSD for multichain targets. Lower is better.
  • structure_key_residue_displacement: mean displacement of configured fixed/key residues. Lower is better.
  • structure_confidence_weighted_rmsd: RMSD weighted by model confidence. Lower is better.
  • structure_ca_deviation_max: maximum per-residue C-alpha deviation. Lower is better.
  • structure_secondary_structure_agreement: coarse secondary-structure agreement. Higher is better.
  • structure_radius_of_gyration: compactness sanity check.
  • structure_exposed_hydrophobics: approximate exposed hydrophobic fraction. Lower is usually better for soluble proteins.
  • structure_clashscore: simple steric clash estimate per 1000 atoms. Lower is better.

Structure Subset Selection

Predicting structures for every decoded sequence can be expensive. Use metrics.structure.selection to evaluate a representative subset:

metrics:
  structure:
    enabled: true
    model: boltz2
    selection:
      enabled: true
      rank_by: ranking_score
      dedupe_sequences: true
      decoded_top_n: 3
      decoded_middle_n: 1
      decoded_bottom_n: 1
      include_controls: [mpnn_reference, mpnn_argmax, wildtype]

The output CSV still includes all sequence rows, but structure columns are filled only for selected rows. Helper columns document selection:

  • structure_metrics_selected
  • structure_metrics_selection_reason
  • structure_metrics_rank

Immunogenicity Metrics

Immunogenicity metrics are disabled by default. When enabled, thyme calls MHCnuggets for overlapping peptides from each sequence and summarizes predicted MHC-I and/or MHC-II binding burden.

Minimum config:

metrics:
  immunogenicity:
    enabled: true
    mode: both          # mhc-i | mhc-ii | both
    alleles_i: [HLA-A02:01, HLA-B07:02]
    alleles_ii: [HLA-DRB101:01, HLA-DRB107:01]

Binder thresholds:

  • strong binder: ic50 <= strong_ic50_threshold
  • weak binder: strong_ic50_threshold < ic50 <= weak_ic50_threshold

Burden scores:

  • immunogenicity_burden_i = (2 * mhc_i_strong_binders + mhc_i_weak_binders) / sequence_length
  • immunogenicity_burden_ii = (2 * mhc_ii_strong_binders + mhc_ii_weak_binders) / sequence_length
  • immunogenicity_burden_score = immunogenicity_burden_i + immunogenicity_burden_ii

Comparison columns:

  • input_immunogenicity_burden
  • delta_immunogenicity_burden_vs_input
  • lower_predicted_immunogenicity_vs_input

Negative delta_immunogenicity_burden_vs_input values indicate lower predicted burden than the input sequence.