Discrepancy between average error calculation from calibration

[gabrielblancogomez]

I was trying to obtain calibration data from participants to drop those with really bad calibration and realized there a discrepancy between calibration data extracted using:

• MNE-Python: Reading from ASC files using read_eyelink_calibration()
• eyelinkio: Reading from EDF files using read_edf().to_mne()

The calibration quality metrics (average error) differ significantly between the two methods, even when reading data from the same recording session. This affects participant exclusion decisions and data quality assessment.

Expected behavior: Both methods should return identical calibration metrics since they're reading from the same eye-tracking session.

Observed behavior: The two methods return different values, leading to inconsistent data quality assessments.

The offsets seems to be the same.

I added here two photos. One from an example participants and another plot which graphs the difference between each function

[scott-huberty]

Hmmm..... 🤔

In MNE-Python's read_eyelink_calibration, we don't calculate the average offset ourselves. Rather, Eyelink's host software calculates the average offset and writes it to the ASCII file, and then we just read that into MNE-Python:

https://github.com/mne-tools/mne-python/blob/2e677b122980a605c1a3a86765f549b70ee0a534/mne/io/eyelink/_utils.py#L818

For example, given an ASCII file contains text like this:

MSG	451371 !CAL VALIDATION HV5 L LEFT  GOOD ERROR 0.29 avg. 0.65 max  OFFSET 0.21 deg. -7.6,-1.9 pix.
MSG	451371 VALIDATE L POINT 0  LEFT  at 960,540  OFFSET 0.12 deg.  0.3,-4.4 pix.
MSG	451372 VALIDATE L POINT 1  LEFT  at 1600,540  OFFSET 0.40 deg.  -10.1,10.9 pix.
MSG	451372 VALIDATE L POINT 2  LEFT  at 320,540  OFFSET 0.12 deg.  -4.5,0.1 pix.
MSG	451372 VALIDATE L POINT 3  LEFT  at 960,720  OFFSET 0.65 deg.  -24.3,-2.9 pix.
MSG	451372 VALIDATE L POINT 4  LEFT  at 960,360  OFFSET 0.35 deg.  -7.3,-10.9 pix.

the average error is 0.29, but if you average together the 5 OFFSET values, you will get .328. So something is off in EyeLink's calculation, or something is off with my understanding of what that avg. number is supposed to represent.

---

Now for EyeLinkIO - I don't think that EyeLink calculates and stores the "average offset" in the EDF file itself, and so we compute the mean ourselves:

eyelinkio/src/eyelinkio/utils/utils.py

Line 172 in 646b9fc

avg_error = np.round(np.mean(offsets), 3)

---

TL;DR: If I manually compute the mean of the set of calibration offsets, I will get the same number regardless of whether I read the calibration with MNE-Python or EyeLinkIO. The discrepancy seems to be in the "average error" number that EyeLink writes to the ASCII files, which MNE reads in and stores as the average offset, e.g.:

MSG	451371 !CAL VALIDATION HV5 L LEFT  GOOD ERROR 0.29 avg. 0.65 max  OFFSET 0.21 deg. -7.6,-1.9 pix.

Interestingly, The Max offset that EyeLink reports is the same number that I would arrive at manually. It's just the average offset that differs.

---

Example:

import eyelinkio

fnames = eyelinkio.utils._get_test_fnames()
fname = fnames[1]
fname

PosixPath('/Users/scotterik/devel/repos/eyelinkio/src/eyelinkio/tests/data/test_raw.edf')

edf = eyelinkio.read_edf(fname)

raw, cals = edf.to_mne()

print(cals[0]["offsets"].mean())
print(cals[0]["avg_error"])

np.float64(0.328)
np.float64(0.328)

Now convert that EDF file to ASCII and read it with MNE:

import mne

cals = mne.preprocessing.eyetracking.read_eyelink_calibration("/path/to/eyelinkio/tests/data/test_raw.asc")

print(cals[0]["offsets"].mean())
print(cals[0]["avg_error"])

np.float64(0.328)
0.29

[scott-huberty]

@gabrielblancogomez right now my best guess is that EDF2ASC calculates "average error" using a trimmed mean:

MSG	451371 !CAL VALIDATION HV5 L LEFT  GOOD ERROR 0.29 avg. 0.65 max  OFFSET 0.21 deg. -7.6,-1.9 pix.

import mne
from scipy import stats

cals = mne.preprocessing.eyetracking.read_eyelink_calibration("/path/to/eyelinkio/tests/data/test_raw.asc")

print( stats.trim_mean(cals[0]["offsets"], 0.25) )
print(cals[0]["avg_error"])

0.29
0.29

I've reached out to SR Research and asked them to clarify. In the mean time maybe you can apply this code snippet to your files and see if that resolves the discrepancy. If so, let me know!

[gabrielblancogomez]

Thanks for checking up on this @scott-huberty!

I thought the MNE function was doing the calculation as well, but that makes sense. Keep me updated about what SR Research says.

I tried applying the trimmed mean instead, but it's still not the same. I attached two graphs: on the left is EyeLinkIO vs. MNE/SR Research average error, and on the right is EyeLinkIO (trimmed) vs. MNE/SR Research avrage error. It actually seems to be even further off. The hue is just whether they would be classified the same using SR's criteria of “good” if avg_error < 1.0 and max_error < 1.5, “fair” if avg_error < 1.5 and max_error ≤ 2.0, and “poor” otherwise. Im not really sure what method they're using, but it likely isn't a trimmed mean.

Personally, I find the EyeLinkIO calculation more intuitive, so I’ll continue using that. So far, I’ve been using it to filter out participants with bad calibration based on the classification above and this discrepancy only seems to affect 2-3 participants, mostly those classified as "good" using the one method and "fair" using the other. The code I'm using to preprocess data and to make this figure is also available here: https://github.com/gabrielblancogomez/q1k_neurosubs/blob/main/code/et/2_gap_et_preprocessing.ipynb

Either way, this is a great initiative, and I would love to contribute to it sometime!

[scott-huberty]

Ahh Ok then I guess I just got lucky that the trimmed mean matched the EDF2ASC calculation in my test file. Will keep you updated 🙏

[scott-huberty]

@gabrielblancogomez here is the response from SR Research:

Hi there - the valildation average is weighted, with the calibration targets having the following weights:

1 2 1
2 4 2
1 2 1

Calibration points are ordered

6 2 7
4 1 5
8 3 9

So, for a five point calibration, ((0.12*4)+(0.40*2)+(0.12*2)+(0.65*2)+(0.35*2))/12 = 0.29.

If you need this implementation I'd accept a PR if you are up for it? We could add some function like:

def weighted_validation_error(offsets):
    offsets = np.array(offsets)
    n = len(offsets)  # 3, 5, 9
    if n not in [3, 5, 9]:
        raise RuntimeError(f"Expected 3, 5, or 9 validation points. Got {n}: {offsets}")
    weights = np.array([4, 2, 2, 2, 2, 1, 1, 1, 1])
    selected_weights = weights[:n]
    weighted_avg = (offsets * selected_weights).sum() / selected_weights.sum()
    return weighted_avg