Skip to content

[Feature Request] TensorView: Support for overlaying Segmentation Masks on 3D Tensors #12010

@HumbleHumbert

Description

@HumbleHumbert

Is your feature request related to a problem? Please describe.
I am using Rerun to visualize medical imaging data (MRI in .nii.gz format). I log the raw MRI volume as a Tensor and visualize it using TensorView to scroll through slices.

However, I also have a brain segmentation mask (also a 3D volume) that I need to validate against the MRI. Currently, TensorView only seems to support visualizing a single tensor at a time. I cannot overlay the segmentation mask on top of the MRI slices as I would in 2D using SegmentationImage.

Describe the solution you'd like
I would like TensorView (or a similar volume slicing view) to support:

  1. Multi-tensor layering: The ability to select multiple tensor entities in the Blueprint.
  2. Segmentation Overlay: Specifically, the ability to overlay an integer-based tensor (mask) on a float-based tensor (image), utilizing AnnotationContext for coloring and opacity.

Describe alternatives you've considered

  • Converting the 3D segmentation to a Mesh3D using marching cubes. This works for the 3D spatial view but does not allow for precise slice-by-slice validation of the segmentation boundaries against the pixel data.
  • Manually "burning" the segmentation into the MRI pixel data before logging (this destroys the original data values).

Additional context
This is a standard feature in medical imaging viewers (like ITK-SNAP or 3D Slicer) and is critical for debugging 3D segmentation models.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions