scifio.save unusable for images with large number of planes

[NicoKiaru]

Hi!

It's an issue with scifio, but it makes labkit unusable if the number of planes is big.
I need to segment files with > 60k planes (200 timepoints, 300 slices, 1 channel). The segmentation is pretty fast, even in cpu mode, however, saving the resulting image is astronomically slow. Each plane (350 x 350 pixels) takes about a second to be saved... This means that my image will take around 17 hours to be saved (around 10 Gb final, thus ~ 170 kb /s !!)

I think, in a situation a bit similar to ome/bioformats#4204, the ifd are fetched way too often.

Here's a visual vm dump during the save operation showing the getifd operation:

[NicoKiaru]

My temporary fix (which requires https://github.com/BIOP/ijp-kheops):

/*-
 * #%L
 * The Labkit image segmentation tool for Fiji.
 * %%
 * Copyright (C) 2017 - 2024 Matthias Arzt
 * %%
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 * 
 * 1. Redistributions of source code must retain the above copyright notice,
 *    this list of conditions and the following disclaimer.
 * 2. Redistributions in binary form must reproduce the above copyright notice,
 *    this list of conditions and the following disclaimer in the documentation
 *    and/or other materials provided with the distribution.
 * 
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
 * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE
 * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
 * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
 * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 * POSSIBILITY OF SUCH DAMAGE.
 * #L%
 */


#@Context context

#@DatasetIOService io

#@DatasetService datasetService

#@StatusService statusService

#@Logger logger

#@File(style = "directory")  input_directory

#@String(value="*.tif") file_filter

#@File(style = "directory") output_directory

#@String(value="_segmentation") output_file_suffix 

#@File(style = "open") segmenter_file

#@Boolean use_gpu

#@TaskService taskService


SegmentationTool segmenter = new SegmentationTool();
segmenter.setUseGpu(use_gpu);
segmenter.openModel(segmenter_file.getAbsolutePath());
segmenter.setProgressWriter(new StatusServiceProgressWriter(statusService));
FileFilter wildcardFileFilter = new WildcardFileFilter(file_filter);
File[] files = input_directory.listFiles(wildcardFileFilter as FileFilter);
Arrays.sort(files);
for (int i = 0; i < files.length; i++) {
	try {
		processFile(segmenter, files, i);
	}
	catch (Exception e) {
		logger.error(e);
	}
}

private <T extends Type<T>> void processFile(SegmentationTool segmenter, File[] files, int i)
	throws IOException
{
	File inputFile = files[i];
	String inputFileName = inputFile.getName();
	String outputFileName = FilenameUtils.getBaseName(inputFileName) + output_file_suffix;
	File outputFile = new File(output_directory, outputFileName);
	if (outputFile.exists()) {
		logger.warn("Labkit: Skipping " + inputFile + " because output file exists already: " +
			outputFile);
		return;
	}
	ImgPlus<?> inputImage = io.open(inputFile.getAbsolutePath()).getImgPlus();
	statusService.showStatus("Labkit segment " + (i + 1) + "/" + files.length +
		": " + inputFileName);
	ImgPlus<T> result = Cast.unchecked(segmenter.segment(inputImage));
	//Dataset dataset = datasetService.create(restult);
	//io.save(dataset, outputFile.getAbsolutePath());
	saveImgPlus(result, outputFile, taskService);
}
	
public static void saveImgPlus(ImgPlus<?> img, File destination, TaskService ts) {
    try {
        String path = destination.getAbsolutePath()+".tiff";//"C:\\kheops\\test_x-" + sizeInPixelX + "_y-"+sizeInPixelY+"_tile-" + tileSize + "_nT-"+nT+".ome.tiff";
        System.out.println("Saving "+path);

        Instant start = Instant.now();

        int nT = (int) img.dimension(3);

        OMETiffExporter.OMETiffExporterBuilder.Data.DataBuilder dataBuilder = OMETiffExporter.builder();

        for (int t = 0;t<nT;t++) {
            dataBuilder.putXYZRAI(0,t,Views.hyperSlice(img,3,t));
        }

        dataBuilder.defineMetaData(img.getName())
                .imageName(img.getName())
                .voxelPhysicalSizeMicrometer(
                        img.axis(0).calibratedValue(1),
                        img.axis(1).calibratedValue(1),
                        img.axis(0).calibratedValue(2))
                .pixelsTimeIncrementInS(img.axis(0).calibratedValue(3))
                .defineWriteOptions()
                .tileSize((int) img.dimension(0),(int) img.dimension(1))
                .monitor(ts)
                .savePath(path)
                .create()
                .export();

        Instant end = Instant.now();

        System.out.println("Export time (ms) \t"+Duration.between(start,end).toMillis());

    } catch (Exception e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}

import io.scif.services.DatasetIOService;
import net.imagej.Dataset;
import net.imagej.DatasetService;
import net.imagej.ImgPlus;
import net.imglib2.type.Type;
import net.imglib2.util.Cast;
import org.apache.commons.io.FilenameUtils;
import org.apache.commons.io.filefilter.WildcardFileFilter;
import org.scijava.Cancelable;
import org.scijava.Context;
import org.scijava.app.StatusService;
import org.scijava.command.Command;
import org.scijava.log.Logger;
import org.scijava.plugin.Parameter;
import sc.fiji.labkit.ui.segmentation.SegmentationTool;
import sc.fiji.labkit.ui.utils.progress.StatusServiceProgressWriter;

import java.io.File;
import java.io.FileFilter;
import java.io.IOException;
import java.util.Arrays;

import ch.epfl.biop.kheops.ometiff.OMETiffExporter;
import loci.common.DebugTools;
import net.imagej.ImageJ;
import net.imagej.ImgPlus;
import net.imglib2.FinalInterval;
import net.imglib2.RandomAccessibleInterval;
import net.imglib2.position.FunctionRandomAccessible;
import net.imglib2.type.numeric.ARGBType;
import net.imglib2.view.Views;
import org.scijava.task.TaskService;

import java.io.File;
import java.time.Duration;
import java.time.Instant;

[maarzt]

@NicoKiaru Thank you very much for researching this problem, for describing it and for the development of a workaround! Sounds like the root of the problem is in SCIFIO or bioformats and will probably be fixed with bioformats 8. Do you think the Labkit source code should be changed?

[NicoKiaru]

I'm not 100% sure, but unfortunately, I fear it's another problem than the one in bio-formats. The bio-formats issue was touching only the pyramidal reader/writer, and the slow down happened only at the very end when the file is closed (writing planes is fast). Here, the slow down happens as every plane is written.