Surface Matrix Test#144
Conversation
Coverage Report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
kaitj
left a comment
kaitj
left a comment
There was a problem hiding this comment.
Going to split this into parts:
- Technical
- Are we considering these to be integration tests (that is the functions we are calling work with real data) or are we planning for these to be part of regression testing (to see if adding / changing features change performance)? If the former, I would leave out all of the outputs proposed to be merged in here and share those offline or in a different repository. If the latter, I would leave out the matrices, but retain the .csv file and perhaps give the output directory a more meaningful name. In either case, I would update the
.gitignorefile appropriately. - A few type casting nitpicks
- Explicitly adding a dependency that is being called (
pandas)
- Are we considering these to be integration tests (that is the functions we are calling work with real data) or are we planning for these to be part of regression testing (to see if adding / changing features change performance)? If the former, I would leave out all of the outputs proposed to be merged in here and share those offline or in a different repository. If the latter, I would leave out the matrices, but retain the .csv file and perhaps give the output directory a more meaningful name. In either case, I would update the
- Scientific (we can also take this discussion offline if it is easier)
- At a glance, the distances seem greater than I would have expected, in particular between the different macaque templates. This may just be a result of the transforms that we are being used, but we should double check to make sure we aren't doing anything unexpected. In any case, we've at least provided a framework now that we can perform these tests and substitute in other values.
- Would also be interesting to have histograms of the distance errors from workbench to gauge how much the vertices have moved
Overall, I think it looks good, just a few changes to be made.
There was a problem hiding this comment.
I think it might make more sense to only to the surface mesh comparison within a given species. I'll check with Ting, but I don't think her cross-species transformation is really meant to produce a human-like surface mesh from a macaque one.
I think NMT2Sym should have better results, so that might indicate that there is a mistake along the way.
kaitj
left a comment
kaitj
left a comment
There was a problem hiding this comment.
Looking good (code-wise). A few minor things to change, and looking at this more, this is a better fit as a regression test
kaitj
left a comment
kaitj
left a comment
There was a problem hiding this comment.
Almost, there. There is also still the set path here for the graph initialization.
Other than that, it might also make sense to wait and potentially incorporate any feedback / changes stemming from @tfunck's previous comment.
|
Thinking about the results of the human to macaque transformation, I think there's a good chance there's a mistake in the human -> macaque transformation because the errors are so much higher than for macaque -> human. Granted the matrix may not be symmetric but I think the error level should be similar between two spaces. Is it possible the wrong transformation file was used for human -> macaque? EDIT: Actually, the above reasoning might not be correct. Ting's xspecies alignment optimizes the functional similarity between the macaque and human brain based on fMRI and doesn't care at all about morphology. If for example a human brain has an extra sulcus that the macaque doesn't, the alignment doesn't try to flatten out the human sulcus when fitting to the macaque. The macaque brains, by contrast, are aligned based on cortical curvature and sulcal depth which is explitly morphological. So I guess it could be possible that the transformation is much worse in one direction versus the other. However, in either case, it doesn't make sense to use the vertex distance (which is a measure of morphological alignment) on transformations that aren't meant to optimize this. So like I was saying before, we should only apply this test within species where morphological transformations have been calculated. |
|
@tfunck here is some log output related to your above comment- fsLR → Yerkes19 Yerkes19 → fsLR So the large error does seem to be symmetry-related. I inspected the relevant files here: does anything stand out to you or @kaitj ? |
|
Nothing at a glance, but if the alignment is based on function rather than morphology than I think it makes sense for the large errors. We should then make a note in the graph edge about this if this is indeed the case. |
|
It could be useful to visualize the transformed surfaces versus their targets. It'll give us a sense of what's going on. |
3 participants
Surface Matrix Error Test (tests/regression/test_surf_matrix.py)
Compute a directed surface-to-surface transform error matrix by measuring median vertex-wise signed-distance error after resampling midthickness cortical surfaces between template spaces using sphere-based barycentric registration.
The resulting matrix captures both directional (asymmetric) and symmetric geometric dissimilarity between brain surface atlases.
Transform error matrix (directed): shows pairwise median surface registration error when mapping each source cortical space (rows) to each target space (columns) using sphere-based resampling, revealing overall compatibility structure across atlases.
Asymmetry matrix: quantifies directional bias in registration by subtracting reverse mappings (A→B − B→A), highlighting strong non-reciprocity particularly for fsLR, where mapping into fsLR is substantially more distorted than mapping out.
Symmetric matrix: summarizes intrinsic geometric dissimilarity between atlas spaces by averaging bidirectional errors ((A→B + B→A)/2), revealing a coherent low-distortion cluster among CIVETNMT, D99, MEBRAINS, NMT2Sym, and Yerkes19, with fsLR as a clear outlier.
The full-scale heatmap shows the complete range of transform errors across all atlas spaces, including fsLR, which allows for a direct comparison of absolute magnitude differences but is visually dominated by high-error outliers. The NHP-scaled heatmap uses a rescaled color limit based on the NPPatlas subset, suppressing fsLR extremes to reveal relative differences within NHP.