fail to process reasonable 3d dataset

[bpavie]

Hi
I tried to process a mrc file but failed:

• The dataset is a mrc file of 1756x1843x143 size, 439MB
• My GPU card is an NVIDIA RTX A4000 with 16GB GPU memory

I used tile_shape parameters to reduce the GPU usage but got this error. Note that without the tile_shape parameter I have the same error

synapse_net.run_segmentation -i 20241017_RTtomo_TS3_25kx_rec.mrc -o 20241017_RTtomo_TS3_25kx_rec_result --tile_shape 128 128 12 -m vesicles_3d
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\u00xxxx\.conda\envs\synapse-net\Scripts\synapse_net.run_segmentation.exe\__main__.py", line 7, in <module>
  File "C:\Users\u00xxxx\Documents\PROJECTS\Leuven\synapse-net\synapse_net\tools\cli.py", line 159, in segmentation_cli
    tiling = parse_tiling(args.tile_shape, args.halo, is_2d=is_2d)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\u00xxxx\Documents\PROJECTS\Leuven\synapse-net\synapse_net\inference\util.py", line 459, in parse_tiling
    default_tiling = get_default_tiling(is_2d=is_2d)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\u00xxxx\Documents\PROJECTS\Leuven\synapse-net\synapse_net\inference\util.py", line 426, in get_default_tiling
    raise NotImplementedError(f"Estimating the tile size for a GPU with {vram} GB is not yet supported.")
NotImplementedError: Estimating the tile size for a GPU with 17.170956288 GB is not yet supported.

Do you have any tips to process it ?
Best
Benjamin

[constantinpape]

I will look into it more later, but what I immediately spot: The tile shape is given in the wrong order. It needs to be passed as ZYX (python axis convention), i.e. like this:

--tile_shape 12 128 128

but the error is likely due to some error issue in our code. I will check it later and report back.

[constantinpape]

Hi @bpavie ,
there was an issue with the inference code, that prevented any GPU with less than 20 GB from working. I fixed this now (though it still won't work with less than 10GB, but I don't think there is much on the market with <10GB anymore).

I fixed this, so once you update the code the prediction should work for you. Let me know if this fixes the issue.
It should also not be necessary to specify the patch shape anymore, the default one should work now.
If you do specify the tile shape, then make sure to give it in ZYX order.

And just fyi: in case you want to segment vesicles in Cryo-ET data, you could also try the model vesicles_cryo.

[bpavie]

Hi @constantinpape

The new version seems to process, but it doesn't seem to not output anything.
I am on Windows 11 and tried:

--output ./output/

--output "./output"

and the full path but none of them output anything.

Example of command:

synapse_net.run_segmentation -i 20241017_RTtomo_TS3_25kx_rec.mrc --tile_shape 12 128 128 --model vesicles_cryo --output "./output/"
Determined tile size: {'x': 256, 'y': 256, 'z': 32}
Processing files:   0%|                                                                                                 | 0/1 [00:00<?, ?it/s]Rescaling the data at 20241017_RTtomo_TS3_25kx_rec.mrc by (1.2522727792913264, 0.8162963302047164, 0.8162963302047164) to match the training voxel size (0.88, 1.35, 1.35)

file structure:

dir
 Volume in drive C is Windows

 Directory of C:\Users\u00xxxx\test

13/12/2024  11:19    <DIR>          .
13/12/2024  11:18    <DIR>          ..
10/12/2024  13:48       460.533.096 20241017_RTtomo_TS3_25kx_rec.mrc
13/12/2024  11:19    <DIR>          output

[constantinpape]

I

13/12/2024  11:19    <DIR>          output

Did you check if there is anything in this directory (output)? The result should be written in there as 20241017_RTtomo_TS3_25kx_rec_prediction.tif.

If that's not the case then we likely have some issue with processing filepaths on windows, as we haven't tested running it on windows yet. I could set up CI later to see if I can reproduce the issu.

[bpavie]

Hi @constantinpape ,
Yes, the output dir is empty:

dir output
 Volume in drive C is Windows
 Volume Serial Number is 920B-93BA

 Directory of C:\Users\u00xxxxx\test\output

13/12/2024  11:19    <DIR>          .
13/12/2024  11:19    <DIR>          ..
               0 File(s)              0 bytes
               2 Dir(s)  83.589.132.288 bytes free

[constantinpape]

Hi @bpavie ,
we have improved the windows installation and I have also added it to the CLI to check that it works.
If you have time, please check out the new installation instructions and see if your issue persists.

If you still have the issue, then it would be great to get a minimal reproducible example, including data to reprocude.