Different sform between native image and derivatives

[jcohenadad]

Data version:

sform is different between the image and its derivatives. Example for sub-P001:

sub-P001_UNIT1.nii.gz

qform_name	Scanner Anat
qform_code	1
qto_xyz:1	0.000000 0.000000 1.000000 -89.557289 
qto_xyz:2	-1.000000 0.000000 0.000000 150.847916 
qto_xyz:3	0.000000 -1.000000 0.000000 120.816551 
qto_xyz:4	0.000000 0.000000 0.000000 1.000000 
qform_xorient	Anterior-to-Posterior
qform_yorient	Superior-to-Inferior
qform_zorient	Left-to-Right
sform_name	Scanner Anat
sform_code	1
sto_xyz:1	-0.000000 -0.000000 1.000000 -89.557289 
sto_xyz:2	-1.000000 -0.000000 -0.000000 150.847916 
sto_xyz:3	0.000000 -1.000000 0.000000 120.816551 
sto_xyz:4	0.000000 0.000000 0.000000 1.000000 
sform_xorient	Anterior-to-Posterior
sform_yorient	Superior-to-Inferior
sform_zorient	Left-to-Right

sub-P001_UNIT1_lesion-manualNeuroPoly.nii.gz

qform_name	Scanner Anat
qform_code	1
qto_xyz:1	0.000000 0.000000 1.000000 -89.557289 
qto_xyz:2	-1.000000 0.000000 0.000000 150.847916 
qto_xyz:3	0.000000 -1.000000 0.000000 120.816551 
qto_xyz:4	0.000000 0.000000 0.000000 1.000000 
qform_xorient	Anterior-to-Posterior
qform_yorient	Superior-to-Inferior
qform_zorient	Left-to-Right
sform_name	Scanner Anat
sform_code	0
sto_xyz:1	0.000000 0.000000 0.000000 0.000000 
sto_xyz:2	0.000000 0.000000 0.000000 0.000000 
sto_xyz:3	0.000000 0.000000 0.000000 0.000000 
sto_xyz:4	0.000000 0.000000 0.000000 1.000000 
sform_xorient	Unknown
sform_yorient	Unknown
sform_zorient	Unknown

This results in overlay issue when visualizing with FSLeyes (which uses sform):

Given that ground truths were made with FSLeyes, I am wondering why we did not observe this issue earlier? Maybe the derivatives data changed at some point?

@Nilser3 how are you able to observe the data and derivatives properly aligned in FSLeyes (eg as in #76)

[Nilser3]

Hi @jcohenadad

I have noticed that:

for image sub-P001_UNIT1.nii.gz

sform_name   Scanner Anat
sform_code   1

It means that the image is aligned with the scanner coordinates and that an affine transformation is required for this alignment.

I see in the image sub-P001_UNIT1_lesion-manualNeuroPoly.nii.gz

sform_name   Scanner Anat
sform_code   0

It means that the image is aligned with the scanner coordinates and that for this alignment there have been no transformations (perhaps that is why they are not aligned with sub-P001_UNIT1.nii.gz).

(see https://nipy.org/nibabel/nifti_images.html)

I am working on the same subject, in the commit ffe427d4d1f62832e5f3567c8ce814eeff9b9764 (last)

sub-P001_UNIT1.nii.gz

sform_name   Scanner Anat
sform_code   1

image sub-P001_UNIT1_lesion-manualNeuroPoly.nii.gz

sform_name   Unknown
sform_code   0

Both images are correctly aligned (see image), does this mean that the image sub-P001_UNIT1_lesion-manualNeuroPoly.nii.gz is aligned in an unknown space and does not need any transformation to be aligned with the image sub-P001_UNIT1.nii.gz?

Then we must verify that the sform and qform must be the same for the mask and image to ensure their alignment.

[jcohenadad]

Ah! It seems that FSLeyes v1.6.1 and v1.5.0 don't deal with overlays in the same way.

On v1.5.0:
The overlay is displayed without warning message, and the overlay is aligned (lesions are visible):

On v1.6.1:
Despite the sform_name/sform_code to be unknown, FSLeyes seems to be using the sform from the overlay and the qform from the image, leading to misalignment (lesion no more visible):

Additional info about the display in FSLeyes, although that doc probably refers to the latest version (1.8.3): https://open.win.ox.ac.uk/pages/fsl/fsleyes/fsleyes/userdoc/display_space.html#display-space.

Sidenote: another cool feature of v1.5.0 is that the JSON sidecar is interpreted and read by FSLeyes:

Whereas it is not on v1.6.1:

Interestingly, this difference in interpreting the sform_name/sform_code is not mentioned in the changelog: https://open.win.ox.ac.uk/pages/fsl/fsleyes/fsleyes/userdoc/changelog.html#thursday-23rd-february-2023

Although now, I am wondering if these differences come from the versions, or the way I access them:

• v1.6.1: from the splashscreen: /Users/julien/miniconda3/share/fsleyes/FSLeyes.app
• v1.5.0: from the Terminal: /Users/julien/fsl/share/fsl/bin/fsleyes

Maybe @pauldmccarthy has some insights for us 😊

Moving forward, I think we should:

• see what's happening with FSLeyes v1.8.3
• modify the sform codes of the image to be NIFTI_XFORM_UNKNOWN (given that there is no reason for using a standard template on these images, which is what the sform is designed for)
• consider modifying the sform matrix of the derivatives to be the same as that of the image (additional security, in case the sform_code is improperly interpreted by a software).

@Nilser3 can you please do that?

[pauldmccarthy]

Hi @jcohenadad @Nilser3, is there any chance that you could share the problematic images (and JSON sidecar files) so I can have a play? There have not been any intentional changes related to the handling of sforms/qforms or sidecar files since version 1.5.0 ....

If you have two images which are voxelwise aligned, but are not aligned when viewed in FSLeyes, a good solution is to use FSL's fslcpgeom command, to copy the sform/qform information from one image to the other. I would recommend having the sform/qform set to something, rather than setting them to NIFTI_XFORM_UNKNOWN, as the latter can lead to confusion w.r.t. distinguishing between left and right.

For clarification, FSLeyes will use either the sform or the qform, depending on the values of the [s|q]form_code - this is detailed under the World coordinate system section of the FSLeyes display space page.

[jcohenadad]

Thank you for chipping in @pauldmccarthy. Here are the image (masked with a 15x15x1 mask centred at 145x195x91 voxel space for ethics reason)-- the mask is on a lesion so you can test the overlay with the lesion mask: sub-P001_UNIT1.zip.

The image does not have a JSON sidecar.

[pauldmccarthy]

Hi @jcohenadad,

Hmm, FSLeyes v1.8.3 seems to be behaving just fine for these images - the image and mask are aligned by default (and FSLeyes doesn't complain about them having different orientations/FOVs):

I took another look at your screenshots, and noticed that FSLeyes is complaining about differing orientations/FOVs in your v1.6.1 screenshot. The code which controls whether this warning is displayed has not changed in several years (it is a simple routine which compares the shape and voxel->world affine of all loaded images), so I'm wondering if you were viewing the same two images in both screenshots?

In FSLeyes 1.8.3, the JSON metadata for the mask is displayed in the info panel for me. One point which I should have made in my previous comment is that JSON information for an image will only be displayed if that image is selected in the overlay list - in your earlier screenshot demonstrating missing JSON metadata, it looks like sub-P001_UNIT1 was selected?

[pauldmccarthy]

One other comment on the use of the sform/qform affines (which I'm sure you're already aware of! but just for completeness :) ) - FSLeyes and FSL use the sform and qform interchangeably, and will use either depending on the values of the [s|q]form_codes, as outlined in the FSLeyes display space page (linked in an earlier comment).

Because FSL uses the sform/qform interchangeably, some FSL tools may "normalise" the sform/qform when writing out an image, setting them both to be the same affine.

However, other tools may follow different conventions, and may use and/or update the sform/qform using different conventions. This is important to take into account for a typical pipeline which uses tools from a variety of software packages. Whenever you invoke any tool which does things differently to the convention you have chosen, you will need to adjust the output images using fslcpgeom or similar.

[jcohenadad]

so I'm wondering if you were viewing the same two images in both screenshots?

yes, here is a screen recording: https://www.dropbox.com/s/kbgrocfb1fkr3e9/Screen%20Recording%202023-09-22%20at%209.15.33%20AM.mov?dl=0

[jcohenadad]

I also confirm that upgrading to 1.8.3 (using pip install --upgrade fsleyes) fixed the issue of overlapping.

However, I am wondering how to launch FSLeyes from the Terminal with a conda installation. I tried launching ./miniconda3/share/fsleyes/FSLeyes.app/Contents/MacOS/fsleyes but got the following errors:

Terminal output

(base) julien-macbook:~ $ ./miniconda3/share/fsleyes/FSLeyes.app/Contents/MacOS/fsleyes 
Traceback (most recent call last):
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/OpenGL/platform/osmesa.py", line 22, in GL
    return ctypesloader.loadLibrary(
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/OpenGL/platform/ctypesloader.py", line 36, in loadLibrary
    return _loadLibraryWindows(dllType, name, mode)
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/OpenGL/platform/ctypesloader.py", line 89, in _loadLibraryWindows
    return dllType( name, mode )
  File "/Users/julien/miniconda3/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: ("dlopen(OSMesa, 0x000A): tried: '/Users/julien/miniconda3/lib/python3.10/lib-dynload/../../OSMesa' (no such file), '/Users/julien/miniconda3/bin/../lib/OSMesa' (no such file), 'OSMesa' (no such file), '/usr/local/lib/OSMesa' (no such file), '/usr/lib/OSMesa' (no such file), '/Users/julien/OSMesa' (no such file)", 'OSMesa', None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/julien/./miniconda3/share/fsleyes/FSLeyes.app/Contents/MacOS/fsleyes", line 5, in <module>
    from fsleyes.filtermain import main
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/fsleyes/__init__.py", line 374, in <module>
    from fsleyes.main import embed, shutdown  # noqa
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/fsleyes/main.py", line 46, in <module>
    import fsleyes.cliserver  as cliserver
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/fsleyes/cliserver.py", line 37, in <module>
    import fsleyes.actions.applycommandline as applycli
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/fsleyes/actions/applycommandline.py", line 28, in <module>
    import fsleyes.parseargs            as parseargs
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/fsleyes/parseargs.py", line 273, in <module>
    import fsleyes.plugins        as plugins
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/fsleyes/plugins/__init__.py", line 241, in <module>
    import fsleyes.views.canvaspanel     as canvaspanel
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/fsleyes/views/canvaspanel.py", line 20, in <module>
    from . import                    colourbarpanel
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/fsleyes/views/colourbarpanel.py", line 15, in <module>
    import fsleyes.gl.wxglcolourbarcanvas as cbarcanvas
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/fsleyes/gl/wxglcolourbarcanvas.py", line 17, in <module>
    import fsleyes.gl.colourbarcanvas as cbarcanvas
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/fsleyes/gl/colourbarcanvas.py", line 21, in <module>
    import OpenGL.GL as gl
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/OpenGL/GL/__init__.py", line 3, in <module>
    from OpenGL import error as _error
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/OpenGL/error.py", line 12, in <module>
    from OpenGL import platform, _configflags
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/OpenGL/platform/__init__.py", line 56, in <module>
    _load()
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/OpenGL/platform/__init__.py", line 53, in _load
    plugin.install(globals())
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/OpenGL/platform/baseplatform.py", line 97, in install
    namespace[ name ] = getattr(self,name,None)
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/OpenGL/platform/baseplatform.py", line 15, in __get__
    value = self.fget( obj )
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/OpenGL/platform/osmesa.py", line 66, in GetCurrentContext
    function = self.OSMesa.OSMesaGetCurrentContext
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/OpenGL/platform/baseplatform.py", line 15, in __get__
    value = self.fget( obj )
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/OpenGL/platform/osmesa.py", line 60, in OSMesa
    def OSMesa( self ): return self.GL
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/OpenGL/platform/baseplatform.py", line 15, in __get__
    value = self.fget( obj )
  File "/Users/julien/miniconda3/lib/python3.10/site-packages/OpenGL/platform/osmesa.py", line 28, in GL
    raise ImportError("Unable to load OpenGL library", *err.args)
ImportError: ('Unable to load OpenGL library', "dlopen(OSMesa, 0x000A): tried: '/Users/julien/miniconda3/lib/python3.10/lib-dynload/../../OSMesa' (no such file), '/Users/julien/miniconda3/bin/../lib/OSMesa' (no such file), 'OSMesa' (no such file), '/usr/local/lib/OSMesa' (no such file), '/usr/lib/OSMesa' (no such file), '/Users/julien/OSMesa' (no such file)", 'OSMesa', None)

Note: I tried inside and outside conda's virtual env.

[pauldmccarthy]

@jcohenadad Depending on how FSLeyes is installed, sometimes the entry points don't get created correctly (this is an issue with conda). You may need to edit <conda-prefix/bin/fsleyes and/or <conda-prefix>/share/fsleyes/FSLeyes.app/Contents/MacOS/fsleyes and change the shebang line to use pythonw or python.app instead of python`. For example, if the shebang line is:

#!/..../miniconda3/bin/python

change it to:

#!/..../miniconda3/bin/pythonw

Regarding the orientation issue, perhaps this is due to a difference in nibabel version - can you check what versions of nibabel and fslpy you have installed for both your 1.5.0 and 1.6.1 FSLeyes installations? They should be listed in the FSLeyes about dialog.

[jcohenadad]

FSLeyes 1.6.1, nibabel [5.1.0], fslpy [3.11.2]

Full output

FSLeyes version: 1.6.1
FMRIB Centre, Oxford, UK
Paul McCarthy
paul.mccarthy@ndcn.ox.ac.uk
FSL version: None
FSL directory: /Users/julien/fsl
OpenGL version: 4.1
OpenGL compatibility: 3.3
OpenGL renderer: AMD Radeon Pro 5300M OpenGL Engine

FSLeyes was developed at the FMRIB Centre, Nuffield Department of Clinical Neurosciences, Oxford University, United Kingdom.

FSLeyes is a Python application which leverages the following open-source software libraries:

 - fsleyes-props [1.9.3] (https://git.fmrib.ox.ac.uk/fsl/fsleyes/props)
 - fsleyes-widgets [0.13.0] (https://git.fmrib.ox.ac.uk/fsl/fsleyes/widgets)
 - fslpy [3.11.2] (https://git.fmrib.ox.ac.uk/fsl/fslpy)
 - indexed_gzip [1.7.1] (https://github.com/pauldmccarthy/indexed_gzip/)
 - IPython [8.12.0] (https://ipython.org/)
 - jinja2 [3.1.2] (http://jinja.pocoo.org)
 - Jupyter notebook [6.5.4] (https://jupyter.org)
 - matplotlib [3.7.1] (http://www.matplotlib.org)
 - nibabel [5.1.0] (http://nipy.org/nibabel)
 - numpy [1.21.6] (http://www.numpy.org)
 - pillow [9.2.0]  (http://python-pillow.org/)
 - pyopengl [3.1.6] (http://pyopengl.sourceforge.net)
 - pyparsing [2.4.7] (http://pyparsing.wikispaces.com/)
 - scipy [1.10.1] (http://www.scipy.org)
 - six [1.16.0] (https://pythonhosted.org/six/)
 - trimesh [3.21.5] (https://github.com/mikedh/trimesh)
 - wxpython [4.2.0] (http://www.wxpython.org)
 - wxnatpy [0.4.0] (https://github.com/pauldmccarthy/wxnatpy/)
 - xnatpy [0.5.1] (https://bitbucket.org/bigr_erasmusmc/xnatpy)

Unfortunately (or fortunately) I don't have access to FSLeyes 1.5.0 anymore because I just upgraded it to 1.8.3.

[jcohenadad]

Depending on how FSLeyes is installed, sometimes the entry points don't get created correctly (this is an issue with conda). You may need to edit <conda-prefix/bin/fsleyes and/or /share/fsleyes/FSLeyes.app/Contents/MacOS/fsleyes and change the shebang line to use pythonw or python.app instead of python`.

I tried it (with both pythonw and python.app), but now getting the following error:

julien-macbook:~ $ ./miniconda3/share/fsleyes/FSLeyes.app/Contents/MacOS/fsleyes 
./miniconda3/share/fsleyes/FSLeyes.app/Contents/MacOS/fsleyes: line 3: import: command not found
./miniconda3/share/fsleyes/FSLeyes.app/Contents/MacOS/fsleyes: line 4: import: command not found
from: can't read /var/mail/fsleyes.filtermain
./miniconda3/share/fsleyes/FSLeyes.app/Contents/MacOS/fsleyes: line 7: syntax error near unexpected token `('
./miniconda3/share/fsleyes/FSLeyes.app/Contents/MacOS/fsleyes: line 7: `    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])'

[pauldmccarthy]

FSLeyes 1.6.1, nibabel [5.1.0], fslpy [3.11.2]

Yep, I think this explains the discrepancy - nibabel has some logic to decide whether a quaternion vector is valid, and thus whether or not to use a NIfTI qform affine. In version 5.1.0 this logic was changed, and this had a knock-on effect in FSLeyes, which I fixed in FSLeyes version 1.7.0. So

• FSLeyes < 1.7.0 and nibabel < 5.1.0 work fine
• FSLeyes < 1.7.0 and nibabel >= 5.1.0 may reject some qforms
• FSLeyes >= 1.7.0 and nibabel == * should be fine

I tried it (with both pythonw and python.app), but now getting the following error:

Looks like the fsleyes entry point is now being interpreted as a shell script. I recall seeing this in the past, but have forgotten the cause. Can you double check that you have a ./miniconda3/bin/pythonw or ./miniconda3/bin/python.app interpreter? Can you also share the contents of the fsleyes file you changed, and which is failing?

[jcohenadad]

Looks like I do have the interpreter:

julien-macbook:~ $ ll ./miniconda3/bin/pythonw 
-rwxrwxr-x  1 julien  staff  135 18 Jan  2023 ./miniconda3/bin/pythonw

julien-macbook:~ $ ll ./miniconda3/bin/python.app 
-rwxrwxr-x  1 julien  staff  135 18 Jan  2023 ./miniconda3/bin/python.app

Can you also share the contents of the fsleyes file you changed, and which is failing?

Here you go:

julien-macbook:~ $ cat ./miniconda3/bin/fsleyes
#!/Users/julien/miniconda3/bin/pythonw
# -*- coding: utf-8 -*-
import re
import sys
from fsleyes.filtermain import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())