exp232 follow-up: why doesn't distal (enhancer) VEP improve with training? — hypotheses to investigate

[gonzalobenegas]

Observation (#232)

The ccre_non_promoter (enhancer) arm is the lone exception to the diagonal-wins + LL-gap↔AUPRC story. Its distal AUPRC sits at ~0.10–0.13 across the entire training trajectory with no learning trend, and distal is the only subset with a ~0 / negative LL-gap↔AUPRC correlation (Pearson −0.10).

Two clues that shape the question

• The constraint signal grows; the AUPRC doesn't. The arm's own val_enhancer LL-gap rises 0.037 → 0.098 over 5000 steps — so it is learning some enhancer constraint — yet distal AUPRC stays flat. The bottleneck is the constraint → distal-VEP transfer, not a total failure to learn. (For scale: cds's val_cds gap goes 0.007 → 0.41.)
• The ccre arm does better off-diagonal on coding than on its own target. From the diagonal it scores splicing 0.238 and synonymous 0.179 — both higher than its own distal 0.127. It is a better splicing predictor than enhancer predictor.

Hypotheses (not committing — full list to investigate)

• H1 — training-set contamination via conservation filtering (lead hypothesis). After conservation filtering, the ccre_non_promoter partition may be enriched for conserved distal cCREs that overlap or border CDS exons (i.e. sit right at the CDS edge), so the arm partly trains on coding sequence. This would directly explain its strong splicing/synonymous off-diagonal and its weak own-distal signal. → Diagnostic: intersect the v4 ccre_non_promoter training intervals with CDS/exon annotations; quantify CDS overlap and CDS-edge proximity. If confirmed, it spins off a v4 data-curation fix.
• H2 — heterogeneous target. ccre_non_promoter mixes enhancers + insulators/CTCF + other distal cCREs → a diffuse training target with no sharp enhancer-specific constraint. → Diagnostic: break the partition down by cCRE subtype; consider an enhancer-only arm.
• H3 — weak / fundamental signal. Enhancers are intrinsically less constrained than coding/splice sites, so there's simply less for a gLM to exploit (gap 0.098 vs cds 0.41). May be a ceiling, not a fixable bug.
• H4 — model too small. 0.25B may lack the capacity for enhancer grammar. → Caveat: Missense VEP degrades with model scale (uniquely among variant classes) — investigate why #279 shows missense AUPRC degrades with scale, so scale isn't uniformly helpful — check enhancers specifically (e.g., exp187's 1B distal arm).
• H5 — train longer. The LL-gap is still rising at step 5000 — do it and the AUPRC keep improving past the budget? → Suspect, since the gap already climbs while AUPRC is flat.
• H6 — eval ceiling / ground truth. The distal mendelian positive set is small (n=58) and effects are diffuse; the ceiling for distal mendelian VEP may just be low. → Diagnostic: compare to exp187 1B distal, a supervised enhancer baseline, or shuffle controls before blaming the gLM.

Cheap first diagnostics when revisited

1. H1 CDS-overlap intersection — most directly testable and, if true, reframes everything.
2. exp187 1B distal — does scale help distal at all? (informs H4.)
3. The ccre arm's own LL-gap-vs-distal-AUPRC scatter — is there any positive regime, or is it flat throughout?

Status

Parking lot — lots of follow-up potential; not committing to a hypothesis. Revisit later.

References: #232 (diagonal + trajectory), #8 (LL-gap↔AUPRC framing), #279 (scale-axis VEP anomaly), #227/#228 (v4 dataset build).

[gonzalobenegas]

Some thoughts:

• For assigning a window to cCRE-non-promoter we could afford dropping any window that overlaps any other type of functional element (since we have so many, most distal)
• We can iterate faster by just training on a single new set of windows for cCRE-non-promoter, and once it works, train on the other 5 regions to see the rest stay consistent

[gonzalobenegas]

🤖 H1 diagnostic — composition of the ccre_non_promoter ("distal") training arm. Measured on the v4 region-labels parquet (region_labels/v4/min0.20.parquet), whose disjoint, priority-resolved *_frac columns give each window's bp overlap with every functional element directly. Reproduction: scripts/issue283/h1_composition.py. n = 427,080 windows.

Overlap with other functional elements (H1):

• 13.4% of windows overlap another functional element (CDS / UTR / ncRNA exon / TSS+5′UTR); the other 86.6% are pure cCRE.
• The overlap is overwhelmingly exonic / CDS: of the non-cCRE bp, CDS = 75%, ncRNA exon = 18%, UTR/TSS = ~7%. Whole-arm, ~10% of windows touch CDS and only 5.4% of the arm's functional bp is non-cCRE; when a window clips CDS it averages 31% coverage. (Such windows span an exon edge → they contain coding sequence and a splice site.)

Genomic context: 66.0% intronic / 20.6% intergenic / 13.4% exon-overlap. (Intronic distal enhancers are legitimate — "distal" = far from the promoter, not intergenic — so the intronic majority is not contamination.)

cCRE-class mix (after conservation filtering): enhancer-like dELS+pELS = 82.9% of cCRE bp (82.0% of windows by dominant class); non-enhancer classes (CA / TF families) = ~17%. Conservation filtering enriches enhancers vs the raw registry (74.6% → 82.9%).

Train vs eval probe: val_enhancer (the LL-gap probe) is built dELS+pELS only with all annotated exons subtracted; the training arm is all non-PLS classes with exons not subtracted — i.e. the probe is a cleaner, narrower population than the arm trains on.

Full per-class breakdown (427,080 windows)

cCRE class	% of windows	bp share
dELS	73.8%	74.0%
pELS	8.9%	8.9%
CA	9.7%	9.0%
CA-CTCF	4.3%	3.6%
CA-H3K4me3	2.9%	2.6%
TF	1.3%	1.0%
CA-TF	0.9%	0.9%