LoRA supervised fine-tuning vs the frozen linear probe on Mendelian-missense VEP (motivated by #341)

[gonzalobenegas]

Status

🔬 Active — verdict converging. Across model scale (46M→476M) and data budget (6–100%), capacity-controlled LoRA FT matches the frozen probe but never beats it — the two scale together (same slope, converging to a tie). With the cost asymmetry (probe = one auto-tuned knob; FT needed a capacity sweep + ~8 knobs to reach parity), the frozen linear probe is the better default for Mendelian-missense VEP on this data budget. But broader data is a lever: a single FT trained on all consequences pooled beats the per-subset probe on the small/weak subsets (3′/5′UTR, ncRNA, distal) while losing on the strong ones — so to beat the probe, spend on more labels, not more FT hyperparameters. Detail: iter2, iter3; findings below.

Question

#341 showed a frozen-embedding linear probe beats the zero-shot LLR on Mendelian missense, by a margin that grows with scale (+0.195 per-chromosome-weighted AUPRC at 4B), because the missense likelihood readout degrades over training while the representation keeps the signal — a cheap linear head recovers it.

Does unfreezing the backbone recover more? Does LoRA supervised fine-tuning — the same labels, the same pooled feature, but the backbone now trainable end-to-end — beat the frozen linear probe on Mendelian-missense per-chromosome-weighted AUPRC? If yes, there is missense signal that is not linearly decodable from the frozen embedding but is reachable by adapting the backbone. If no, the frozen probe is the ceiling for this label budget and the lever is more labels, not more trainable parameters.

This is EDA — a bounded look at the readout landscape, no preregistered hypothesis. Output is insights (does FT beat the probe, does the answer depend on scale, is it an overfitting story), not a locked leaderboard. Train split only; the test split is held for a future pre-registered final pass.

Honest prior — expect this to be hard. #180 found supervised heads on pooled embeddings did not materially beat zero-shot, and #314 explicitly deferred fine-tuning ("MLP/XGBoost/LoRA — #180 found no lift; revisit only if linear plateaus"). A positive, scale-growing result would be the notable finding; a null almost certainly means we couldn't out-regularize the data scarcity (next section), not that the backbone lacks signal.

The central risk — overfitting is the whole investigation

With ~4,910 training missense SNVs (491 positives) after holding out chr1, and a LoRA-adapted 46M–4B backbone carrying 10³–10⁶ trainable parameters, the model will memorize the training set by default — the Phase-0 overfit-100 check expects train AUPRC → 1.0. The frozen probe (≈ few-thousand params, hard L2, StandardScaler) is a strong baseline precisely because it can't overfit. So this investigation has exactly one failure mode and one open question, and they are the same thing:

Can we regularize the fine-tune hard enough that it generalizes to the held-out chromosome better than the frozen probe — or does every recipe overfit past the probe's generalization?

Therefore the exploration is organized around regularization as the primary sweep axis and the train-vs-validation generalization gap as the primary instrument — logged at every step of every run, not just at the end.

The diagnostic is the trajectory vs the probe baseline line. For every run we log validation-chromosome AUPRC per training step against the constant frozen-probe baseline (the number to beat): does the trajectory climb above the probe before it inevitably turns down (overfitting), or plateau below it (nothing to gain)? Early stopping picks the peak; the height of that peak over the probe is the result. (We can optionally warm-start the head from the fitted probe so the curve departs exactly from the probe line — but the comparison does not depend on it; see Architecture.)

Anti-overfitting levers, ranked (what we actually sweep):

lever	mechanism	priority
Training length / early stopping	fewer steps = closer to the frozen init; the dominant regularizer here. One run's val-vs-step curve gives the whole axis for free.	#1
LoRA capacity — rank (1/2/4/8/16), adapted-layer depth (top-K layers only), target modules (attn ⊂ attn+mlp)	fewer trainable params = less to overfit; top-layers-only also targets the degraded readout (#341's mechanism) while leaving the representation frozen	#2
Weight decay on LoRA	pulls adapters toward 0 = toward the frozen model; a direct "don't move far" penalty	#3
Dropout (LoRA + head)	standard variance reduction; may need to go well above the 0.05 default	#3
Learning rate	lower LR + early stop = smaller, implicitly-regularized moves from init	#4 (couples with #1)
Data augmentation — RC strand (in), optional ±few-bp window jitter (variant stays in-window)	expands effective sample count cheaply	supporting
Batch size	SGD-noise-as-regularizer (smaller) vs cleaner early-stop signal (larger); genuinely unclear at this data scale	exploratory / secondary

Meta-risk — don't overfit the selection. The early-stop/hyperparameter signal is a single held-out chromosome (noisy — chr3 ≈ 71 missense positives). Searching a large grid against one noisy val chrom overfits the selection. Mitigations: keep each sweep coarse and staged (vary one lever family at a time, not a full grid), rotate / inner-CV the val chrom, and treat any single-fold win as provisional until the full nested-LOCO confirms it.

Scope

In: LoRA supervised fine-tuning (missense only), the matched Mendelian train split, the scaling ladder (subset of the 8 rungs), per-chromosome-weighted AUPRC, single-fold dev + full nested-LOCO honest eval, baselines = frozen probe + zero-shot LLR.

Out (deferred): full-FT / non-LoRA PEFT; external / larger labeled sets; non-missense consequence classes; complex / SGE / QTL datasets; self-supervised readout-repair; test-split final pass; productionizing into evals_v2.

Approach

Baselines to beat (same rows, same metric, paired)

The frozen probe and LLR recompute per-fold directly from the existing S3 scores parquets — the scaling-v0.5-*/mendelian_traits.parquet files already carry emb_ref/emb_alt (from the #341 embed overlay) and llr_fwd/llr_rc. No GPU needed for the baselines.

size	frozen probe	zero-shot LLR	Δ (probe − LLR)
46M	0.425	0.444	−0.019
76M	0.438	0.479	−0.041
128M	0.485	0.489	−0.004
255M	0.522	0.487	+0.035
476M	0.534	0.454	+0.080
1B	0.553	0.428	+0.125
2B	0.577	0.389	+0.188
4B	0.586	0.391	+0.195

(#341 numbers; per-chromosome-weighted AUPRC on missense, train split.) The frozen probe column is the number LoRA FT must beat at each scale.

Data

bolinas-dna/evals_mendelian_traits @ 4aed58e5 (PR #194 k=9 rebuild), train split, missense_variant subset only:

• 5,800 SNVs / 580 positives (exactly 1:9; matched-pair — match groups pit a positive against 9 nearest-neighbor negatives on (consequence, chrom) strata + continuous TSS/exon-distance features, so this is an across-gene, not within-gene, task).
• 12 train chromosomes: {1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, X} (odd + X, by upstream split construction; even + Y = held-out test, never loaded). CV just groups by the chrom column (like traitgym_nested_oof's LeaveOneGroupOut) — there is no in-repo odd/even filter to replicate.
• Leave-out-chr1 dev fold: train on ~4,910 missense (491 pos), eval on chr1's 890 missense / 89 positives — enough positives for a usable dev signal, few enough to keep iteration fast (but noisy — a single-fold point estimate, treat as directional).

Architecture (siamese — mirrors the probe's feature exactly)

A shared LoRA-adapted backbone processes the ref and alt 255 bp+BOS windows; each is entire-window mean-pooled (fp32, BOS excluded — the #341 pooling); the feature is [pool_ref, pool_alt − pool_ref] (the probe's concat_ref_delta); a single linear head → logit, trained with BCE. Same input construction as the frozen probe — the only difference is that the backbone (via LoRA) is now trainable. RC: reverse-complement as train-time augmentation (random strand per example — free regularization, biologically motivated), FWD+RC pooled-average at eval (matches the probe + pipeline's two-pass scoring).

Head init — fresh head; probe kept as an external baseline line (default). The probe head is calibrated to the FWD+RC-averaged embedding the pipeline manufactures — and these are causal, unidirectional models, so that "bidirectionality" is entirely the strand-average (+ mean-pool), not something a single forward produces. We therefore do not warm-start from the probe by default: we train a fresh linear head and keep the frozen probe as a constant baseline line to clear. Optional diagnostic: warm-starting the head (fold StandardScaler in; LoRA Δ=0) makes the eval curve depart exactly from the probe line at step 0 — but that init is only honest if training also consumes the averaged feature (otherwise the probe head is fed single-strand inputs it wasn't calibrated for); since we keep RC as a train-time augmentation, the warm-start stays an add-on, not a dependency. Deferred representation arm: a causal model's more native per-window summary is the last-token hidden state (attended over the whole context) rather than the probe's mean-pool + strand-average — possibly a better FT target, but it breaks apples-to-apples with the probe, so not iter-1.

What we reuse vs build

Reuse (no rework): Genome window reads (src/marin_dna/data/genome.py), the variant-window + RC + SNV tokenization (transform_llr_clm / _get_variant_window, src/marin_dna/data/transforms.py), the HF load pattern AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True) (src/marin_dna/pipelines/evals/inference.py), the entire-window pooling span [n_prefix, n_prefix+window_size) (src/marin_dna/model/scoring.py), the frozen-probe recompute traitgym_nested_oof / pair_feature_from_bundle (src/marin_dna/pipelines/evals/variant_probe.py), and the scorer per_chrom_weighted_ap / per_chrom_ap_table (src/marin_dna/pipelines/evals/metrics.py) verbatim.

Build (there is zero training/PEFT code in the repo today): add peft as a dep; a siamese LoRA classifier module + training loop + chromosome-grouped CV harness, all in a new tested src/marin_dna/pipelines/finetune/ (library home per Research Code Values). Run drivers + SkyPilot YAML in scripts/. This branch won't merge to main (research; cited via commit-pinned permalinks).

CV & leak-proofing (the discipline — hyperparameters stay exploratory)

• Train split only. Even-chrom test never touched.
• Dev fold (fast loop): hold out chr1 as the fold's test; hold out one more train chromosome (e.g. chr3) as a validation chrom for early stopping and hyperparameter selection; train on the other 10. Report AUPRC on chr1. chr1 is never peeked at for selection (mirrors nested-LOCO).
• Honest final: full nested leave-one-chromosome-out over all 12 train chroms (inner val chrom per fold for early stopping), OOF predictions concatenated and scored once as per-chromosome-weighted AUPRC, with paired cluster-bootstrap deltas vs the frozen probe and LLR on identical rows.

Fixed defaults (everything not being swept)

AdamW; fresh linear head (frozen probe kept as an external baseline line, not the init); 10% warmup → cosine; bf16 (+ gradient checkpointing ≥1B); plain BCE (pos_weight a later knob); ≥3 seeds, report mean ± SD (small-data FT is high-variance); RC augmentation in training, FWD+RC pooled-average at eval; per-step train + val-chrom AUPRC logged for every run (the main instrument). Starting points for the swept regularization levers (see the levers table above — seeds for the sweep, not settled values): LoRA r = 8 / α = 16 / dropout = 0.05, attention q/k/v/o, LoRA lr 1e-4 / head lr 1e-3, weight decay 0, early-stop patience 3 on val-chrom AUPRC with a generous max-epoch cap.

Metric

Per-chromosome-weighted AUPRC on missense (per_chrom_weighted_ap, the #341 / #331 / TraitGym primary), paired vs frozen probe and vs LLR (paired bootstrap resampling match_group on the full CV). Single-fold dev = AUPRC on chr1's missense (one number; report seed spread).

Plan (order of exploration)

• Phase 0 — de-risk + instrument (46M, cheap). Confirm gradients flow through LoRA; overfit-100 sanity (train AUPRC → ~1.0). Wire the per-step train + val-chrom AUPRC logging that is the investigation's main instrument. Confirm the recomputed frozen-probe baseline line matches Scaling ladder × Mendelian VEP: linear probe vs zero-shot LLR through the official pipeline (#320/#331) #341 on the chr1 fold (the number to beat); if the optional warm-start is enabled, check it reproduces that AUPRC at step 0. Recompute the frozen-probe + LLR baselines per-fold from the S3 parquets (no GPU) — the exact numbers to beat.
• Phase 1 — can we out-regularize the data scarcity? (255M, leave-out-chr1). 255M: probe clearly beats LLR (0.522 vs 0.487) yet iterates fast. Start with ONE warm-started run at defaults and read the val-AUPRC-vs-step trajectory — the single most informative run: does val ever exceed the frozen probe, and where does it peak vs. overfit? Then stage the regularization sweep to lift that peak above the probe, one lever-family at a time (avoid selection-overfitting), 3 seeds each: (1) training length — free from the trajectory; (2) LoRA capacity — rank ↓ and top-K-layers-only; (3) weight decay + dropout ↑; (4) LR ↓. Select on the val chrom; report chr1 AUPRC vs probe. Gate: if no regularization lifts val above the probe, that's a clean "can't out-regularize the scarcity at this scale" — decide whether large scale (more probe-over-LLR headroom) is worth a test before calling it, and flag the deferred more-labels arm as the implied lever.
• Phase 2 — scale trend (best recipe, leave-out-chr1). Run the Phase-1 winner across {46M, 128M, 255M, 1B, 4B} (light per-model LR retune). Overlay three curves — LLR / frozen probe / LoRA FT — and read whether the FT-vs-probe gap grows, shrinks, or vanishes with scale.
• Phase 3 — honest full CV (winners only). Full nested LOCO (12 folds) on the 2–3 most informative scales; per-chrom-weighted AUPRC + paired bootstrap vs probe & LLR. Train split only.

Current findings

iter1 (255M, leave-out-chr1, missense, single seed). Baselines: frozen probe 0.523, zero-shot LLR 0.489 (LLR full-12 recompute = 0.487 = #341 ✓). The default LoRA (rank 8) overfits from ~epoch 5 (val-loss bottoms epoch 4, train-loss → 0.06 by epoch 11) and scores 0.500 — below the probe, at the LLR floor. The dominant lever is capacity: rank 8→2 or top-4-layers both lift it to 0.524 ≈ the probe, while weight-decay (0.510 @ 0.1) / LR barely move it. ⇒ with capacity control the fine-tune matches the frozen probe at 255M but does not clearly beat it. Point-per-config table + methodology in the iter1 comment.

iter2 — scale + data-scaling (comment). FT vs probe vs LLR across 46M→476M (chr1 missense): 46M .379/.362/.367, 76M .422/.415/.442, 128M .480/.505/.474, 255M .524/.523/.489, 476M .545/.548/.472 — FT tracks the probe at every scale, no crossover (matches it; ≫ LLR at large scale). Data-scaling (255M, subsample): FT and probe rise together and converge — FT has no steeper slope, so more data along this axis won't let it overtake. ⇒ LoRA FT matches but never beats the frozen probe, across scale and data budget; the probe wins on simplicity/robustness (1 auto-tuned knob vs ~8). Remaining: pooled-all-subsets (broader data, evaluated per subset).

Open questions / next steps

• Now: trace FT vs frozen-probe vs zero-shot-LLR across the scaling ladder (46M → 476M) with the capacity recipe (low rank), single seed — does the FT-vs-probe relationship (a tie at 255M) change where the probe-over-LLR gap is large? Small rungs included, to watch the pattern evolve; per-scale chr1 baselines computed alongside.
• Compute: ≤476M on the A10G (no 4B / A100 yet); torch.compile + gradient accumulation to hold the effective batch constant as batch shrinks on bigger rungs.
• Deferred: seed-averaging (once a scale shows a real edge — single-seed for now); ≥1B / 4B; full-FT / non-LoRA PEFT; larger external labeled set (ClinVar / AlphaMissense) if data scarcity caps the gain; other consequence classes / datasets; self-supervised readout-repair (the Scaling ladder × Mendelian VEP: linear probe vs zero-shot LLR through the official pipeline (#320/#331) #341 alternative framing); test-split final pass.

Tracking

Description = current state. Comments = append-only 🤖 iteration log with commit-pinned permalinks to code. Pipeline README (if any) = how-to-run only. References: #341 (motivator), #314 (frozen-probe protocol + the "defer FT" note), #180 (supervised-head null), #302 / #306 (the missense readout-degradation origin).

---

🤖 Opened by Claude Code as an agent-research EDA investigation, motivated by #341.

[gonzalobenegas]

🤖 Design refinement (methodology, pre-runs): head initialization.

The warm-start idea — init the FT head from the fitted frozen probe so step 0 ≡ the probe — implicitly assumes the model feeds the head a FWD+RC-averaged embedding. But these are causal, unidirectional models; that "bidirectionality" is manufactured by the pipeline's strand-average (+ window mean-pool), not produced by any single forward. So a probe-initialized head is a clean init only if training also consumes the averaged feature — which conflicts with keeping RC as a train-time augmentation (an overfitting lever we want to keep).

Change: the default is now a fresh linear head, with the frozen probe kept as an external baseline line to clear; the warm-start is demoted to an optional diagnostic (honest only under averaged-feature training). RC-augmentation stays. The core comparison — does the val-AUPRC-vs-step trajectory clear the probe line? — is unchanged; it never needed the warm-start, because the "starts on the probe" property lives on the eval curve (eval averages FWD+RC at init, where LoRA Δ=0), not on the backbone's directionality.

Also filed a deferred representation arm: the last-token hidden state as a more causal-native per-window summary than the probe's mean-pool + strand-average (breaks apples-to-apples with the probe, so not iter-1).

Body updated accordingly: The central risk, Architecture, Fixed defaults, Plan → Phase 0.

[gonzalobenegas]

🤖 iter1 — harness on PyTorch Lightning, and the 255M lever sweep: capacity is the knob; LoRA FT matches the frozen probe but doesn't beat it at 255M.

Harness (built + validated on GPU, A10G)

Rebuilt on PyTorch Lightning + wandb (project gonzalobenegas/dna-issue369). Code at 6ec6fe0 (harness) + run_finetune.py + recompute_chr1_baselines.py. Methodology evolved from the plan:

• Head init — fresh head, probe as an external baseline line (the warm-start assumed FWD+RC-averaged embeddings, but these are causal/unidirectional models — see the head-init refinement). RC-augmentation in training, FWD+RC pooled-average at eval.
• Early-stop on val-loss (patience 2); reported checkpoint selected by best val-AUPRC — val-AUPRC on ~71 val positives is too noisy to stop on, val-loss is smooth.
• One combined backbone forward (stack ref+alt / the 4 eval strands) for throughput.
• Memory: batch 64 OOMs 255M on the 24 GB A10G → batch 32 + eval-batch 64 + expandable_segments.

Baselines — chr1 (leave-out-chr1, missense)

	chr1 AUPRC
frozen probe	0.523
zero-shot LLR	0.489

Recompute path validated: LLR full-12 = 0.487, matches #341's 255M LLR exactly.

255M lever sweep (single seed, point-per-config)

config (255M)	chr1 AUPRC	Δ vs default
rank 2	0.524	+0.024
top-4-layers	0.524	+0.024
weight-decay 0.1	0.510	+0.010
weight-decay 1.0	0.502	+0.002
default (rank 8, wd 0)	0.500	—
lower LR (3e-5)	0.498	−0.002

Read. The default LoRA overfits from ~epoch 5 (val-loss bottoms epoch 4, train-loss collapses to 0.06 by epoch 11). The dominant lever is capacity: cutting trainable params (rank 8→2, or adapting only the top 4 layers) lifts it to 0.524 ≈ the probe (0.523), while weight-decay / LR barely move it. So with capacity control the fine-tune matches the frozen probe — but does not clearly beat it here.

Scope update — pivot to the scale axis

255M is not the discriminating regime: the probe-over-LLR gap there is only ~0.034, ≈ single-seed noise (the default itself moved 0.488→0.500 on a selection tweak). The phenomenon is a scale story (#341: probe-over-LLR +0.125 @1b, +0.195 @4B). So next I'll trace FT vs probe vs LLR across the scaling ladder (46M → 476M) with the capacity recipe — including the small models, to watch the pattern evolve. Single seed for now (pattern-finding, not noise-nailing); ≤476M on the A10G (no 4B / A100 yet). When batch must shrink on bigger rungs I'll accumulate gradients to hold the effective batch constant across scales.

Results parquets under s3://oa-bolinas/analysis/issue369/.

[gonzalobenegas]

🤖 Scope addition — a data-scaling law for fine-tuning.

New axis (a data axis, not another hyperparameter): subsample the training set and measure chr1 AUPRC vs training-set size, for FT vs the frozen probe. This directly tests the core hypothesis — is FT capped by the ~5k-example scarcity?

• If FT's curve is steeper (more data-hungry) and would cross the probe's if extrapolated → more labels would let FT overtake the probe, which justifies the deferred external-dataset arm (ClinVar / AlphaMissense).
• If both plateau together → the probe just wins on this task, and more data won't change the verdict.

Implemented as --train-frac (label-stratified subsample, keeps the ~10% positive rate) at 8b0520b. Running at a fixed model where the probe clearly beats the LLR (255M), across fractions {12.5, 25, 50, 100%}, for both FT and the probe.

Context for this addition: the scale sweep (46M→255M) shows FT ≈ the probe within single-seed noise (no clean crossover), so "does FT beat the probe" hinges on whether it's data-limited — hence this axis.

[gonzalobenegas]

🤖 iter2 — the scale + data-scaling verdict: capacity-controlled LoRA FT matches the frozen probe but never beats it, across model scale and data budget.

FT vs frozen-probe vs zero-shot-LLR across scale (46M → 476M, chr1 missense)

rung	FT (rank 2)	probe	LLR
46M	0.379	0.362	0.367
76M	0.422	0.415	0.442
128M	0.480	0.505	0.474
255M	0.524	0.523	0.489
476M	0.545	0.548	0.472

FT tracks the probe at every scale, within ±0.02–0.03 single-seed noise — no crossover. Even at 476M, where the probe beats the LLR by +0.076, FT lands exactly on the probe (0.545 vs 0.548) and far above the LLR. End-to-end fine-tuning recovers the probe's signal but doesn't exceed it, anywhere.

Data-scaling (255M): does more data break the ceiling?

Subsampling the ~5k missense train set, FT and probe on the same fractions:

# train	FT	probe	LLR
~300	—	0.416	0.489
~600	0.425	0.451	0.489
~1200	0.465	0.475	0.489
~2400	—	0.516	0.489
~4900	0.524	0.523	0.489

FT and the probe rise together and converge — FT has no steeper slope (the probe is even slightly more data-efficient at low N). Extrapolating, more data along this axis wouldn't let FT overtake; they plateau together. (Both clear the LLR only past ~1.5k examples — itself a nice illustration of how weak the zero-shot missense readout is.)

Verdict (so far)

Across both axes — model scale and data budget — LoRA FT matches the frozen probe and never beats it. With the cost asymmetry (the probe is one auto-tuned knob — nested-CV L2; FT needed a capacity sweep + ~8 fussy knobs just to reach parity), the honest read is that the frozen linear probe is the better tool for this task on this data budget — equal accuracy, far simpler and more robust.

One test remains (data-scope note): pooled-all-subsets FT (~2.8× data, ~16k pooled) evaluated per subset vs the per-subset probe — whether broader data (esp. for the small subsets) changes it. Running now; I'll append.

Methodology (this round)

• PyTorch Lightning + wandb; val-loss early-stop (patience 2) + best-val-AUPRC checkpoint; one combined backbone forward.
• Grad-accum verified — 476M (batch 8 × accum 4) logs global_step 131/263/395/… = 132 optimizer-steps/epoch = effective batch 32, matching the batch-32 rungs.
• 476M fit on a 24 GB A10G via gradient checkpointing + eval-batch 16 — no A100.
• Probe grids on a separate many-core CPU box (probe_grid.py, n_jobs=-1).
• Code + figures at 97db778; result parquets under s3://oa-bolinas/analysis/issue369/.

[gonzalobenegas]

🤖 iter3 — pooled-all-subsets (the "broader data" test): it does break the ceiling, but only where data was the binding constraint.

One FT (rank 2) trained on all consequence subsets pooled (~16k variants vs 5.8k missense), evaluated per subset vs the per-subset probe (255M, chr1):

subset	pooled-FT	per-subset probe	winner
3′UTR	0.265	0.125	pooled-FT +0.14
5′UTR	0.278	0.187	pooled-FT +0.09
ncRNA-exon	0.196	0.150	pooled-FT +0.05
distal	0.500	0.480	pooled-FT +0.02
missense	0.447	0.523	probe −0.08
splicing	0.457	0.531	probe −0.07
tss-proximal	0.155	0.255	probe −0.10
synonymous	0.384	0.697	probe −0.31

Pooling helps the small / weak-signal subsets (3′/5′UTR, ncRNA, distal — pooled-FT beats the per-subset probe) and hurts the strong ones (missense 0.524→0.447; synonymous collapses 0.697→0.384). This is #314's per-subset-vs-pooled tension resolving both ways: for data-starved subsets, the extra data outweighs the signal-dilution cost of a shared head; for strong / specialized subsets — synonymous especially, whose signal is very distinct — the shared head can't specialize and loses badly.

Refined verdict

• On the headline missense question, the frozen probe still wins (pooling even hurts missense — dilution).
• But "broader data" is a genuine lever: pooled FT beats the probe on the data-starved subsets. So the iter2 "FT never beats the probe" holds only where the probe already had enough data.
• Actionable read: the way to make fine-tuning beat the frozen probe is more labels (weak subsets / a real external set), not more FT hyperparameters — consistent with the data-scaling result (FT is data-limited, and the probe's one-knob simplicity wins at equal data).

(476M pooled-all still running — grad-ckpt is slow; I'll append if it shifts anything.)

[gonzalobenegas]

🤖 476M pooled-all confirms the pattern is robust across scale. Same "helps weak, hurts strong" split: pooled-FT beats the per-subset probe on ncRNA-exon (0.388 vs 0.195), 3′UTR (0.430 vs 0.294), 5′UTR (0.300 vs 0.287); loses on missense (0.511 vs 0.548 — pooling again drops it below the missense-only 0.545), splicing (0.479 vs 0.534), tss (0.162 vs 0.276), and synonymous (0.420 vs 0.752). So the pooled-subsets effect holds at both 255M and 476M.