Sign disagreement between published QTL feathers and Supp. Table 4 for caqtl_microglia and dsqtl_yoruba

[dryingpaint]

Sign disagreement between published QTL feathers and Supp. Table 4 for caqtl_microglia and dsqtl_yoruba

Hi! While reproducing the QTL coefficient evaluation against your paper's Suppl. Table 4 numbers, I ran into an inconsistency that I think is most likely a feather export bug, but it could also be that there's a documented sign-correction step I'm missing. Wanted to flag it either way.

Summary

For 3 of the 5 caQTL/dsQTL coefficient datasets you publish at gs://alphagenome/evals/, the signed pearsonr(prediction, target) computed directly on the feather matches the paper's Suppl. Table 4 value. For 2 datasets (caqtl_microglia and dsqtl_yoruba), the magnitude matches but the sign is opposite.

Dataset	Direct from feather	Suppl. Table 4
caqtl_african	+0.7367	+0.7368 ✅
caqtl_european	+0.5914	+0.5916 ✅
caqtl_smc	+0.6870	+0.6870 ✅
caqtl_microglia	−0.6354	+0.6357 ❌
dsqtl_yoruba	−0.8323	+0.8323 ❌

Reproducer (no install needed, runs on any machine with pandas + scipy)

import os
import pandas as pd
from scipy.stats import pearsonr
from urllib.request import urlretrieve

BASE = "https://storage.googleapis.com/alphagenome/evals/"
cases = [
    ("caqtl_african",   "caqtl_african_variant_coefficient_human_predictions"),
    ("caqtl_european",  "caqtl_european_variant_coefficient_human_predictions"),
    ("caqtl_smc",       "caqtl_smc_variant_coefficient_human_predictions"),
    ("caqtl_microglia", "caqtl_microglia_variant_coefficient_human_predictions"),
    ("dsqtl_yoruba",    "dsqtl_yoruba_variant_coefficient_human_predictions"),
]
for label, name in cases:
    local = f"/tmp/{name}.feather"
    if not os.path.exists(local):
        urlretrieve(BASE + name + ".feather", local)
    df = pd.read_feather(local)
    r, _ = pearsonr(df["prediction"], df["target"])
    print(f"{label:<20} pearsonr(prediction, target) = {r:+.4f}")

Output:

caqtl_african        pearsonr(prediction, target) = +0.7367
caqtl_european       pearsonr(prediction, target) = +0.5914
caqtl_smc            pearsonr(prediction, target) = +0.6870
caqtl_microglia      pearsonr(prediction, target) = -0.6354
dsqtl_yoruba         pearsonr(prediction, target) = -0.8323

What I checked

• The paper distinguishes signed vs unsigned Pearson explicitly (e.g. caQTL Fig. 5d: "Signed Pearson r = 0.74; unsigned Pearson r = 0.45"), so Suppl. Table 4's pearsonr column is clearly the signed version — the magnitudes match perfectly across all 5 datasets, only the sign disagrees on 2.
• I separately verified by running model.predict_variant from this SDK on a few caqtl_microglia variants — the values reproduce the feather's prediction column to within bf16 numerics (r ≈ 0.999). So the prediction column is faithful to model output; the apparent disagreement is between target and the paper number.
• The other 3 datasets process through the same loading code I'm using (target → effect_size, prediction → score) and they line up with the paper exactly. So this isn't something on my end being applied inconsistently.

Question

Is there a per-dataset sign-correction step in your table-generation pipeline that the published feathers don't reflect (e.g. a polarity flip based on which allele was assigned REF in the upstream QTL study for those two)? Or is the target column for caqtl_microglia and dsqtl_yoruba simply exported with the wrong sign?

Either is fine — either the feathers should be re-exported, or the convention should be documented somewhere users can find it. Right now anyone naively running pearsonr(prediction, target) on the published artifacts gets the opposite sign of the paper for those two datasets.

Happy to send a PR with whatever fix you prefer (re-sign the feather column, or add a documentation note + a small apply_sign_convention() utility).

Why this matters downstream

Anyone benchmarking against your numbers using gs://alphagenome/evals/ directly hits this — for instance our radical-eval pipeline cross-published baselines for all 5 datasets and the microglia/dsqtl_yoruba ones came out with the wrong sign, traceable entirely to this.

Thanks!

[pgrosu]

Hi Melissa (@dryingpaint),

It's understandable, but here are a few things that might help:

$1)$ Did you have a chance to look at the following Jupyter notebook?

You'll notice the following notes:

    'dsqtl_yoruba_variant_coefficient_human': {
        'output_type':      'DNASE',
        'metric_name':      "pearsonr_mean_track_aggregation_ontology_curie:['EFO:0002784']",
        'metric_fn':        lambda x: pearsonr(x['target'], x['prediction']).statistic,
        'reported_metric':  0.8323,
        'notes':            'The sign of the correlation flips depending on which allele is labelled REF vs. ALT.',
    },
    'caqtl_microglia_variant_coefficient_human': {
        'output_type':      'DNASE',
        'metric_name':      "pearsonr_mean_track_aggregation_ontology_curie:['CL:0000862']",
        'metric_fn':        lambda x: pearsonr(x['target'], x['prediction']).statistic,
        'reported_metric':  0.6357,
        'notes':            'The sign of the correlation flips depending on which allele is labelled REF vs. ALT.',
    },
...
Eval: dsqtl_yoruba_variant_coefficient_human
  Reported:   0.8323
  Recomputed: -0.8323
  Notes:      The sign of the correlation flips depending on which allele is labelled REF vs. ALT.

Eval: caqtl_microglia_variant_coefficient_human
  Reported:   0.6357
  Recomputed: -0.6354
  Notes:      The sign of the correlation flips depending on which allele is labelled REF vs. ALT.

For reference, EFO:0002784 refers to GM12878 (lymphoblastoid cells) and CL:0000862 refers to suppressor macrophage $-$ which in this case are microglia.

$2)$ Regarding the Pearson correlation function, the paper states that the same Pearson R correlation function is used: "Signed Pearson R correlation uses raw values; unsigned Pearson R uses absolute values first". Since these predict chromatin accessibility for different variants, the correlation can be taken on the actual predicted values (REF vs ALT), or the absolute value of |REF vs ALT|. One measures direction, and the other magnitude similarity. Usually the absolute value is not taken when performing Pearson correlation $-$ as it distorts the distribution of the data $-$ but in this case it works for measuring relative effect size correlation between the predicted and observed variants regarding chromatin accessibility.

$3)$ Regarding the selection of REF and ALT, the Supplementary information to the paper states:

1. Allele-Specific Predictions AlphaGenome generates two sets of genomic predictions centered on
the variant: one for the DNA sequence containing the REF allele and another for the sequence with the
ALT allele. The REF allele sequence used for comparison is either the standard reference genome allele
or, when specified, the specific REF allele defined by the evaluation dataset.

Since chromatin accessibility uses a center-mask (a fixed-width window) variant scoring strategy, the rest of the calculation is then based on the following:

Hope it helps,
Paul