Lately, ABBA is slow on opening projects from samba server

[carlocastoldi]

Hi Nico, it's still me: your worst enemy!

I am trying to understand why, from v0.10.*, ABBA on Windows seems a lot slower than usual to open a save state.
However I am not sure the problem arises from ABBA: it might be from the samba configuration of our SSH server.

Internally, we save all of our images onto a Linux server exporting a samba interface. This allows us to have fast access both from Windows and Linux.
When opening a (cached) project from ABBA, it takes nothing. However when opening a project from Windows, it fails to recover the bfmemo files and it takes ~10 minutes to regenerating them (creating the ABBA project on Linux or Windows seems to have no impact). It's quite frustrating.

Following is one of the errors displayd when opening one image from Windows:

ZeissCZIReader initializing \\TENIBRE\bs\users\Maddalena\Viral Injections\data\images\215\an215_20250227_298.czi
[ERROR] temp file rename returned false: \\TENIBRE\bs\users\Maddalena\Viral Injections\data\images\215\.an215_20250227_298.czi.bfmemo1704020308618151887
[WARN] deleting invalid memo file: \\TENIBRE\bs\users\Maddalena\Viral Injections\data\images\215\.an215_20250227_298.czi.bfmemo
com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 115
Serialization trace:
lasers (loci.formats.in.OIRReader)
readers (loci.formats.ImageReader)
reader (loci.formats.DimensionSwapper)
reader (loci.formats.FileStitcher)
helper (loci.formats.in.FilePatternReader)
readers (loci.formats.ImageReader)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:159)
	at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:758)
	at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:117)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:130)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:877)
	at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:356)
	at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:299)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:796)
	at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:124)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:130)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:796)
	at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:124)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:130)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:796)
	at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:124)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:130)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:796)
	at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:124)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:130)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:877)
	at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:356)
	at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:299)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:796)
	at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:124)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:130)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:774)
	at loci.formats.Memoizer$KryoDeser.loadReader(Memoizer.java:169)
	at loci.formats.Memoizer.loadMemo(Memoizer.java:924)
	at loci.formats.Memoizer.setId(Memoizer.java:702)
	at ch.epfl.biop.bdv.img.bioformats.BioFormatsOpener.getNewReader(BioFormatsOpener.java:516)
	at ch.epfl.biop.bdv.img.bioformats.BioFormatsOpener$ReaderPool.createObject(BioFormatsOpener.java:714)
	at ch.epfl.biop.bdv.img.bioformats.BioFormatsOpener$ReaderPool.createObject(BioFormatsOpener.java:690)
	at ch.epfl.biop.bdv.img.ResourcePool.acquire(ResourcePool.java:96)
	at ch.epfl.biop.bdv.img.bioformats.BioFormatsOpener.<init>(BioFormatsOpener.java:238)
	at ch.epfl.biop.bdv.img.qupath.QuPathOpener.<init>(QuPathOpener.java:198)
	at ch.epfl.biop.bdv.img.opener.OpenerSettings.create(OpenerSettings.java:395)
	at ch.epfl.biop.bdv.img.OpenersImageLoader.lambda$createOpeners$2(OpenersImageLoader.java:151)
	at java.util.Arrays$ArrayList.forEach(Arrays.java:3880)
	at ch.epfl.biop.bdv.img.OpenersImageLoader.createOpeners(OpenersImageLoader.java:149)
	at ch.epfl.biop.bdv.img.OpenersImageLoader.<init>(OpenersImageLoader.java:100)
	at ch.epfl.biop.bdv.img.XmlIoOpenersImageLoader.fromXml(XmlIoOpenersImageLoader.java:96)
	at ch.epfl.biop.bdv.img.XmlIoOpenersImageLoader.fromXml(XmlIoOpenersImageLoader.java:47)
	at mpicbg.spim.data.generic.sequence.XmlIoBasicImgLoader.fromXml(XmlIoBasicImgLoader.java:80)
	at mpicbg.spim.data.generic.sequence.XmlIoAbstractSequenceDescription.fromXml(XmlIoAbstractSequenceDescription.java:123)
	at mpicbg.spim.data.generic.XmlIoAbstractSpimData.fromXml(XmlIoAbstractSpimData.java:186)
	at mpicbg.spim.data.generic.XmlIoAbstractSpimData.load(XmlIoAbstractSpimData.java:118)
	at mpicbg.spim.data.generic.XmlIoAbstractSpimData.load(XmlIoAbstractSpimData.java:88)
	at sc.fiji.bdvpg.spimdata.importer.SpimDataFromXmlImporter.apply(SpimDataFromXmlImporter.java:75)
	at sc.fiji.bdvpg.spimdata.importer.SpimDataFromXmlImporter.get(SpimDataFromXmlImporter.java:68)
	at sc.fiji.bdvpg.scijava.adapter.AbstractSpimdataAdapter.deserialize(AbstractSpimdataAdapter.java:126)
	at sc.fiji.bdvpg.scijava.adapter.AbstractSpimdataAdapter.deserialize(AbstractSpimdataAdapter.java:57)
	at com.google.gson.internal.bind.TreeTypeAdapter.read(TreeTypeAdapter.java:76)
	at com.google.gson.Gson.fromJson(Gson.java:1227)
	at com.google.gson.Gson.fromJson(Gson.java:1329)
	at com.google.gson.Gson.fromJson(Gson.java:1300)
	at com.google.gson.internal.bind.TreeTypeAdapter$GsonContextImpl.deserialize(TreeTypeAdapter.java:179)
	at sc.fiji.bdvpg.scijava.adapter.source.SpimSourceAdapter.deserialize(SpimSourceAdapter.java:93)
	at sc.fiji.bdvpg.scijava.adapter.SourceAndConverterAdapter.deserialize(SourceAndConverterAdapter.java:180)
	at sc.fiji.bdvpg.scijava.adapter.SourceAndConverterAdapter.deserialize(SourceAndConverterAdapter.java:58)
	at com.google.gson.internal.bind.TreeTypeAdapter.read(TreeTypeAdapter.java:76)
	at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:40)
	at com.google.gson.internal.bind.ArrayTypeAdapter.read(ArrayTypeAdapter.java:72)
	at com.google.gson.Gson.fromJson(Gson.java:1227)
	at com.google.gson.Gson.fromJson(Gson.java:1329)
	at com.google.gson.Gson.fromJson(Gson.java:1271)
	at sc.fiji.bdvpg.services.SourceAndConverterServiceLoader.run(SourceAndConverterServiceLoader.java:108)
	at ch.epfl.biop.atlas.aligner.MultiSlicePositioner.loadState(MultiSlicePositioner.java:1141)
	at ch.epfl.biop.atlas.aligner.command.ABBAStateLoadCommand.run(ABBAStateLoadCommand.java:27)
	at org.scijava.command.CommandModule.run(CommandModule.java:196)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:125)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:64)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:247)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
[WARN] file deletion failed \\TENIBRE\bs\users\Maddalena\Viral Injections\data\images\215\.an215_20250227_298.czi.bfmemo

Contrarily, on Linux it spits no error and opens it in 2 seconds:

Creating slice an215_20250214_261.czi - Scene #01...
Slice an215_20250214_261.czi - Scene #01 created!
Creating slice an215_20250214_261.czi - Scene #02...
Slice an215_20250214_261.czi - Scene #02 created!
Creating slice an215_20250214_261.czi - Scene #03...
Slice an215_20250214_261.czi - Scene #03 created!
Creating slice an215_20250214_261.czi - Scene #04...
Slice an215_20250214_261.czi - Scene #04 created!
Creating slice an215_20250214_261.czi - Scene #05...
Slice an215_20250214_261.czi - Scene #05 created!
Creating slice an215_20250214_261.czi - Scene #06...
Slice an215_20250214_261.czi - Scene #06 created!
Creating slice an215_20250214_261.czi - Scene #07...
Slice an215_20250214_261.czi - Scene #07 created!
Creating slice an215_20250214_261.czi - Scene #08...
Slice an215_20250214_261.czi - Scene #08 created!
Creating slice an215_20250214_262.czi - Scene #05...
Slice an215_20250214_262.czi - Scene #05 created!
Creating slice an215_20250214_262.czi - Scene #07...
Slice an215_20250214_262.czi - Scene #07 created!
Creating slice an215_20250214_262.czi - Scene #06...
Slice an215_20250214_262.czi - Scene #06 created!
Creating slice an215_20250214_262.czi - Scene #08...
Slice an215_20250214_262.czi - Scene #08 created!
Creating slice an215_20250214_263.czi - Scene #5...
Slice an215_20250214_263.czi - Scene #5 created!
Creating slice an215_20250214_263.czi - Scene #6...
Slice an215_20250214_263.czi - Scene #6 created!
Creating slice an215_20250214_263.czi - Scene #7...
Slice an215_20250214_263.czi - Scene #7 created!
Creating slice an215_20250214_264.czi - Scene #6...
Slice an215_20250214_264.czi - Scene #6 created!
Creating slice an215_20250214_266.czi - Scene #2...
Slice an215_20250214_266.czi - Scene #2 created!
Creating slice an215_20250214_266.czi - Scene #4...
Slice an215_20250214_266.czi - Scene #4 created!
Info:Current X Angle : .000 has been updated to -3.526
Current Y Angle : .000 has been updated to 6.973

Resolving output success in command ch.epfl.biop.atlas.aligner.command.ABBAStateLoadCommand

What do you think? Is this something that could be due to a problem with our IT configuration? I am of the impression that some operations (e.g. renaming?) are not permitted on the server

[NicoKiaru]

When opening a (cached) project from ABBA, it takes nothing. However when opening a project from Windows

What's opening a cached project vs opening a project from Windows ?

[carlocastoldi]

I would guess that—independently of the OS—when *.bfmemo files are missing or invalid, the project can be considered not cached because ABBA has to regenerate them. The step of regenerating bfmemo files takes a while.

On windows i would assume that the project is cached if bfmemo files are existing in the image folder.

[NicoKiaru]

Reviving this quite late, but this does sound like this issue, no ?

It's a pain point specific to big czi files + windows and samba.

Maybe you can activate the Quick Start CZI Reader in Fiji, but even if it fixes performance on Fiji side, I fear the pain will reappear when opening in QuPath.

[carlocastoldi]

I am revising old emails that I must have missed. Thank you for your patience!

The fact that we had this discrepancy between windows and linux tells me that this issue had nothing to do with how BioFormats opens CZI files.
Unfortunately I won't be back into the lab until the january 12th, where I have access a windows machine connected to a SMB server. I'll let you know asap!

[carlocastoldi]

Hi Nico, I am back on this.
I have tested it again and I am of the impression that the issue has something to do with the permissions of the sambda server exposed to windows vs those for linux.

Additionally, i think ABBA, when this happens, tries to regenerate bfmemo files multiple times (x7 times, to be precise). After which, it gives up and it moves on.
This makes opening a project with the images on a drive with no permission for "renaming" very slow because it didn't cache them. And it is even slower because it generates the bfmemo for 7 times and then fails saving it to file.

abba-linux.log
abba-win1.log
abba-win2.log
abba-win3.log
abba-win4.log
abba-win5.log
abba-win6.log
abba-win7.log

[carlocastoldi]

I confirm this is what happens, because if I manually remove the old .bfmemo and open the ABBA project on Windows, it generates them without asking the drive for a "renaming" operation. This made the project be opened a lot faster, even if the images weren't cached yet.

Obviously once the .bfmemo generated by Windows are there, they are recognised as "good ones" by Windows and the next time it's even faster. Just like it is usually on Linux.

[carlocastoldi]

I think the problem arises from here:
https://github.com/ome/bioformats/blob/8ba547afad0d419516ba30728c8a8127356f3f01/components/formats-bsd/src/loci/formats/Memoizer.java#L1030

Indeed, java.io.File.renameTo behaves differently on the two systems. On Windows i see that it does not rename the file, if it already exists.

The documentation states:

Many aspects of the behavior of this method are inherently platform-dependent: The rename operation might not be able to move a file from one filesystem to another, it might not be atomic, and it might not succeed if a file with the destination abstract pathname already exists. The return value should always be checked to make sure that the rename operation was successful. As instances of File are immutable, this File object is not changed to name the destination file or directory.
Note that the Files class defines the move method to move or rename a file in a platform independent manner.

however, running the any of the following lines works as expected on Windows (i.e. overwrites the existing files on the samba drive):

Files.move(path_old, path_existing, StandardCopyOption.REPLACE_EXISTING)
Files.move(path_old, path_existing, StandardCopyOption.ATOMIC_MOVE)
Files.move(path_old, path_existing, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE)

I wonder what sun.nio.fs.WindowsPath does different with move from java.io.File.renameTo... 🤔

[carlocastoldi]

Oh the problem appears to be how samba "translates" linux permissions on Windows.
Even with 0777 file permissions, Windows has no "Modify" permissions (different from "Write" permissions).

This is something I nor the IT admin of my institute knows how to fix, as of now.

[NicoKiaru]

Maybe it's worth asking on the forum if the location of bfmemo files can be configured ? I vaguely remember that this was discussed.

[carlocastoldi]

I don't think there is an ~easy way.
QuPath appears to have an option that specifies an alternative path for .bfmemo files.
By default, otherwise, it creates a temporary qupath-memo-XYZ folder that is then given to OME's Memoizer.
That same directory is then emptied once QuPath is closed.

If i understand correctly, this means that, by default, QuPath doesn't keep .bfmemo files across launches. I suppose that's because for them it's not a big deal to cache them every time 🤷‍♂️
Perhaps ABBA has different requirements, tho.

However, I wonder why .bfmemo produced Linux and Windows are not compatible.
I also wonder whether, if the BioFormats memoization option was given to QuPath, the .bfmemo files produced by QuPath would be fully compatible with ABBA (and vice-versa).

[NicoKiaru]

QuPath appears to have an option that specifies an alternative path for .bfmemo files.
By default, otherwise, it creates a temporary qupath-memo-XYZ folder that is then given to OME's Memoizer.
That same directory is then emptied once QuPath is closed.

That's great!

I suppose that's because for them it's not a big deal to cache them every time

Weird, creating bfmemo can be costly, but I can see how this prevents issues, like write persmissions not enabled or differences in bio-formats versions causing conflicts in the memo file.

Perhaps ABBA has different requirements, tho.

Not really, but speed on opening can be important, as you can see in this issue.

However, I wonder why .bfmemo produced Linux and Windows are not compatible.

It's partly because some file formats reader are serializing the full path of the file. Path on windows =/= path on mac => issue. (for instance ome/bioformats#3957)

[carlocastoldi]

Perhaps ABBA has different requirements, tho.

Not really, but speed on opening can be important, as you can see in this issue.

However, in this case, the issue is that when ABBA has no rights to overwrite the .bfmemo files, it is even slower because it generates them 7 times and fails 7 times to write it to file before giving up.

It's partly because some file formats reader are serializing the full path of the file. Path on windows =/= path on mac => issue. (for instance ome/bioformats#3957)

Ooh I see, then. Yes! It could work only if it saved relative paths, not absolutes.