OpenEXR IO and compression performance for small tile sizes

[lji-ilm]

Summary

OpenEXR supports different tile sizes in its file's data layout. A smaller tile size provides better granularity of the image data, facilitate localized pixel read/write with less overhead of unwanted pixels. A larger tile size, when compression is involved, provides larger, uninterrupted encoding & decoding contexts and therefore facilitates better performance when the entire image is being read or written.

It is well-expected that performance of read/write entire images would be lower when the tile size is small, compared to larger tiles or scanline data layouts. (For the scanline data layout, each compression encoder can decide on their own how much data it takes in as one continuous encoding context). But it is unclear how much performance degradation would be introduced by shrinking the tile size, and it is also unclear that how much the compression ratio would suffer when tile sizes are reduced.

Here, I provide an easy-to-reproduce experiment to measure these factors. We demonstrate that the performance degradation of whole image read/write is linearly correlated to the tile's diameter, and the compression ratio also took a hit when the tile size is small. See the "Results" section.

Experiment setup

One should have both oiio and exr 's binary tools under the current PATH of the command line. Specifically, the following tools are used in this experiment:

1. oiiotool
2. exrmaketiled
3. exrinfo
4. exrmetrics

The following experiments are done using a fresh build of openEXR 3.4.4 and OpenImageIO 3.1.8.

First, create a 4K half-RGBA exr image with scanline data layout and no compression. The image's content is white noise. Let's also verify the file's data layout that it is exactly what we would like.

$ oiiotool --pattern noise:type=uniform:min=0:max=1 4096x2160 4 -d half --compression none -o whitenoise_4k.exr
$ exrinfo -v whitenoise_4k.exr
File 'whitenoise_4k.exr': ver 2 flags longnames
 parts: 1
 part 1: <single>
  Software: string 'OpenImageIO 3.1.8.0 : 1FC23AF3783083E5519C95F8D3DD1B975C666586'
  capDate: string '2026:01:09 11:51:37'
  channels: chlist 4 channels
   'A': half samp 1 1
   'B': half samp 1 1
   'G': half samp 1 1
   'R': half samp 1 1
  compression: compression 'none' (0x00)
  dataWindow: box2i [ 0, 0 - 4095 2159 ] 4096 x 2160
  displayWindow: box2i [ 0, 0 - 4095 2159 ] 4096 x 2160
  lineOrder: lineOrder 0 (increasing)
  pixelAspectRatio: float 1
  screenWindowCenter: v2f [ 0, 0 ]
  screenWindowWidth: float 1

The next step is using the following bash script to convert this scanline image into all the tile sizes we would like to test -- that is, from 8 to 1024, on every power of 2.

#!/bin/bash

for size in 8 16 32 64 128 256 512 1024; do
    echo "Making tile size: $size"
    exrmaketiled -t "$size" "$size" -z none "whitenoise_4k.exr" "whitenoise_tiled${size}.exr"
done

Note that all files generated have no compression, so they're all approximately the same sime of around 70Mbs (2bytes per sample, 4 sample per pixle, 4096x2160 pixles).

Then, we run exrmetrics on all the result files, including the original whitenoise_4k.exr which represents the scanline data layout.

#!/bin/bash

for size in 8 16 32 64 128 256 512 1024; do
    echo "Benchmarking tile size: $size"
    echo "Starting time:"
    date +"%Y-%m-%d %H:%M:%S.%3N"
    exrmetrics -z all "whitenoise_tiled${size}.exr" > "result_tile${size}.json"
done

echo "Benchmarking scanline exr"
echo "Starting time:"
date +"%Y-%m-%d %H:%M:%S.%3N"
exrmetrics -z all "whitenoise_4k.exr" > "result_scanline.json"

On my AMD Ryzen 5 PRO 4650U laptop under WSL2 in Windows 11, the output timing is the following. Note that this experiment is single thread.

Benchmarking tile size: 8
Starting time:
2026-01-09 14:35:58.220
Benchmarking tile size: 16
Starting time:
2026-01-09 14:54:02.882
Benchmarking tile size: 32
Starting time:
2026-01-09 14:58:49.307
Benchmarking tile size: 64
Starting time:
2026-01-09 15:00:25.546
Benchmarking tile size: 128
Starting time:
2026-01-09 15:01:09.968
Benchmarking tile size: 256
Starting time:
2026-01-09 15:01:38.774
Benchmarking tile size: 512
Starting time:
2026-01-09 15:02:03.815
Benchmarking tile size: 1024
Starting time:
2026-01-09 15:02:25.136
Benchmarking scanline exr
Starting time:
2026-01-09 15:02:45.905

One can see that, without going into details, that the tile8 benchmark run took a whooping 18 minutes to finish! Oops!

Results

I used some jupyter notebook to collage all json files into one csv file, which is also uploaded in this issue page. Then, I summoned some Google Gemini AI power and worked with the bots for the following visualization.

Read timing

First, let's plot the average "read time" of all compression types against the tile size.

It almost look like logarithmic to me. Let's change the vertical axis into log scale.

Now, the horizontal axis are power of 2s in terms of the tile's diameter (8x8, 16x16, 32x32, etc). A straight line in log scale means the performance curve in general is still linear to the tile's diameter, and sublinear to (linear to the sqrt of) the number of pixel inside the tile.

Next, let's break down the performance by compression types:

Also changing the vertical axis to be logarithmic:

There are some behaviours amongst the compression types, but in general everyone still follows the previous observation. One can see that these lines are mostly flat, and evenly spaced on a log scale chart.

The most curious part here, however, is that the none compression type still follows the performance degradation curve against the tile size. There are no codec involved when there is not even a compression, but why the read-whole-image operation still slows down so much when the tile size getting smaller? This seems to indicate that the performance degradation for smaller tile sizes at least have a factor in the exr framework itself, instead all to blame on the compression codecs.

Compression ratio

Now let's take a look at compression ratio. In these plots, the compression ratio is defined as the uncompressed size divided by the compressed size, so the compression is more efficient (better) when the ratio is larger. Here is the average compression ratio plotted against tile size:

Here is a detail view of all compression types' compression ratio performance against tile size:

There are quite some impressive jiggles here and there that may worth technical deep dives, if some compression types are of special interest of a particular studio or project. There also seems to be a bug somewhere in the pipeline that did not report rle's compression ratio and thus rle become a flat 1.0 line on the bottom, same as none compression. But one general observation is that tile size does significantly impact compression ratio performance, on average to almost 20%.
Even the b44 compression, which is supposed to be fixed rate, end up being slightly worse when there are many tiles due to the overhead.

Take aways

The performance degradations are expected, but was larger than what I thought. In particular, the read performance of small-tiled images follows the diameter of the tile for all compression types, or even when there is no compression involved. 18+ mins to finish exrmetrics single run on a 70MB 4K image sounds quite slow for a modern standard, even considering this experiment is single thread. The compression ratio, on average, can also take a 20% hit when the tile size is small (e.g. 8x8).

My result csv file

Note that rle's output size seems to all be identical to the original uncompressed file size. This might be a bug in the pipeline or it's because the source is white noise, so Run Length Encoding performs particularly bad on it.

exr_tiled_exrmetric_results.csv

[lgritz]

Some interesting data..

Instead of exrmetrics, I tried just using oiiotool to read the whole image:

#!/bin/bash

if [[ ! -e whitenoise_4k.exr ]] ; then
oiiotool --pattern noise:type=uniform:min=0:max=1 4096x2160 4 -d half --compression none -o whitenoise_4k.exr
for size in 8 16 32 64 128 256 512 1024; do
    echo "Making tile size: $size"
    name="whitenoise_tiled${size}.exr"
    oiiotool "whitenoise_4k.exr" --tile $size $size -o $name
done
fi

for size in 8 16 32 64 128 256 512 1024; do
    name="whitenoise_tiled${size}.exr"
    /usr/bin/time oiiotool --threads 1 -i:now=1:native=1 $name -echo "{TOP.tile_width}"
done

I rigged it for single threaded (--threads 0 will use as many threads as cores, if you want to try that, too), and the -i:now=1:native=1 causes the file to be immediately and fully read, in the native format (no default translation to float internally).

The results are... interesting... and different than what you got. I'm not seeing that dramatic a difference in read time based on tile size. I mean, there is a little -- 8x8, and to a lesser extent, 16x16, are more expensive, as expected, because of the extra overhead and read calls. But for 32x32 and up, the tile sizes all seem to have very similar performance.

8
        1.30 real         0.99 user         0.29 sys
16
        0.91 real         0.76 user         0.14 sys
32
        0.86 real         0.74 user         0.11 sys
64
        0.84 real         0.73 user         0.10 sys
128
        0.82 real         0.71 user         0.10 sys
256
        0.86 real         0.75 user         0.10 sys
512
        0.89 real         0.77 user         0.11 sys
1024
        0.92 real         0.76 user         0.14 sys

Mac laptop. Haven't tried Linux yet, but you can try it using the script above. Can anybody else reproduce this? The above are for compression 'none', but results for zip aren't super different.

[lji-ilm]

My experiments are mostly under WSL2 since i don't have mac or linux outside of work.
If the performance was like what you reported the degradation seems in line with expectation. I can change the title to include wsl setup.

[ThiagoIze]

WSL2 is going to have poor file read performance if the data is stored on ntfs instead of the ext4 linux filesystem. Smaller tiles means more file IO requests and could explain what you see when file IO is slow. @lji-ilm where did you have your files stored?

[peterhillman]

The "reread time" is also interesting here: that's when exrmetrics reads files from a memory buffer not from the filesystem. It does that to avoid complications like operating-system level caching of files in memory. The reread times don't show such a variation in performance. That suggests this particular filesystem is just not good at many small reads, even if they are sequential in the file.

It could be worth backporting exrmetrics to an older version of the library that used the C++ read path, and confirming this behavior is consistent, rather than a performance regression introduced by switching the read path to the Core library.

[lji-ilm]

WSL2 is going to have poor file read performance if the data is stored on ntfs instead of the ext4 linux filesystem. Smaller tiles means more file IO requests and could explain what you see when file IO is slow. @lji-ilm where did you have your files stored?

The files are in WSL native partition, ~/ASWF. They are not under the outside Windows 11 FS e.g. they're not at /mnt/c/...

[lji-ilm]

The "reread time" is also interesting here: that's when exrmetrics reads files from a memory buffer not from the filesystem. It does that to avoid complications like operating-system level caching of files in memory. The reread times don't show such a variation in performance. That suggests this particular filesystem is just not good at many small reads, even if they are sequential in the file.

It could be worth backporting exrmetrics to an older version of the library that used the C++ read path, and confirming this behavior is consistent, rather than a performance regression introduced by switching the read path to the Core library.

I agree that this experiment is more of measuring the behaviour of openEXR's OS API calls rather than the internal compute code of openEXR itself, and particularly, a strange behaviour of WSL2 under windows 11.
That being said I thought the behaviour these OS API calls are quite important, as they affect the end-to-end pipeline performance seen by a studio stakeholder anyway. If EXR was calling OS API in a particularly unperformant sequence, then I thought we would treat that too. Either we change our sequence or report it to the OS maintainers.

As Larry pointed out this almost perfect log plot probably is a WSL bug. I'll verify it a bit more with a few Linux native setups I have access to. The compression ratio plot should hold true regardless of the WSL anomaly.

[lji-ilm]

I can reproduce @lgritz 's result on WSL. The oiio whole read doesn't seems to be extremely slow on tile 8.
Running Larry's script on Windows 11 WSL2 Ubuntu 24 gives:

8
0.06user 0.30system 0:00.40elapsed 92%CPU 
16
0.02user 0.13system 0:00.17elapsed 92%CPU
32
0.02user 0.09system 0:00.13elapsed 92%CPU
64
0.02user 0.08system 0:00.11elapsed 94%CPU 
128
0.01user 0.10system 0:00.12elapsed 94%CPU 
256
0.02user 0.08system 0:00.12elapsed 92%CPU
512
0.02user 0.10system 0:00.13elapsed 93%CPU 
1024
0.02user 0.11system 0:00.14elapsed 92%CPU 

[lji-ilm]

I also run the original experiment (in the post, using exrmetrics) on my work computer, which is native Alma Linux and have many CPU cores and enough RAM for a car this year. The test image is still the white noise 4K patch turned into a tile size 8x8, no compression exr. I made sure the image is on a local drive and not on a network location.

Interestingly I can reporduce the slowness.
Here are results on native Alma Linux

$exrinfo whitenoise_88.exr 
File 'whitenoise_88.exr':
  type: 'tiledimage'  compression: 'none'
  tiles: size 8 x 8 level 0 (single image) round 0 (down)
  displayWindow: [ 0, 0 - 4095 2159 ] 4096 x 2160
  dataWindow: [ 0, 0 - 4095 2159 ] 4096 x 2160
  channels: 4 channels
   'A': half samp 1 1
   'B': half samp 1 1
   'G': half samp 1 1
   'R': half samp 1 1
  tiled image has levels: x 1 y 1
    x tile count: 512 (sz 4096)
    y tile count: 270 (sz 2160)
$/usr/bin/time exrmetrics -z all whitenoise_88.exr
493.56user 21.51system 8:37.45elapsed 99%CPU 

SO this is another 8.5 minutes run!

I have attached exrmetrics's output here. This is quite similar to the result i got from WSL. And the slowest part are the write operations of piz, dwaa and dwab.

exrmetrics.json

I feel i missed this point (of writing time) in the original post. Nevertheless i think if this log curve is reproducible across platforms then it worth more effort to dig in.

Also I don't understand why exrmetrics read disk file much slower than oiiotool? Since by Larry's script it's like 0.06 sec to read through oiio but with exrmetrics it's always 2 seconds +? @peterhillman

[lgritz]

I have some new results coming soon, too, as I made a new test that isolates the writes and reads outside of oiiotool, to get more to the pure costs.

Will update today if I can.

[lgritz]

I added read and write vs tile size benchmarks to some OIIO unit tests (imageinout_test, for those who follow such things), directly using OIIO::ImageInput and ImageOutput -- no ImageBuf, no oiiotool. In theory, the lowest overhead way you can read and write images via the OIIO APIs.

Here are my Linux results, for both write and read, multithread and single thread. (The image is 4k x 2k, RGBA half. All times are in ms. I used zip compression for both, I think.)

tile size	exr write (ms)	exr read (ms)	1 thread->	write	read
scanline	30.6	5.1		311.8	84.2
4	919.0	402.0		3671.9	1272.9
8	252.3	98.7		1559.1	488.3
16	78.6	27.6		838.5	220.3
32	42.4	12.3		555.1	118.1
64	32.5	8.2		415.4	89.3
128	33.1	6.7		362.1	81.6
256	41.0	9.1		331.4	79.6
512	64.3	15.5		319.1	80.4
1024	95.4	30.0		315.9	84.1
2048	165.6	50.5		319.3	94.9

Some summary comments:

• Write is 2-5x slower than read (as expected -- compression is always much slower than decompression).
• Tiny tiles are much more expensive, as we would expect. Stay away from 4x4 or 8x8 tiles!
• OIIO uses 64x64 tiles by default for textures -- and thankfully, read and write times, if not optimal, were not far off.
• Performance in the middle range of tiles -- say, between 32x32 and 512x512 -- was relatively flat versus tile size.
• Very large tiles started to get expensive again for the multithread case, but that's because it's just not able to parallelize as much. You can see that it's much less pronounced for single threaded.

I didn't post here the numbers for my Mac laptop, but the overall character was pretty similar.

Also, I did similar benchmarks for TIFF, and the overall character was also very similar. OpenEXR always writes much faster (often ~2x) than TIFF. Exr reads slightly slower than TIFF (~10-20%) for single threaded, but is faster (20%+) for the multithreaded case, because the underlying libtiff is not as good as openexr under threading (more statefulness, longer-held locks). This is a pleasant change -- until fairly recent versions, openexr tile reads were signficantly slower than exr tile reads.

I will post a PR to add these tests to the OIIO repo in the coming days (it needs to land behind another PR in progress that fixes some bugs I uncovered in the process of writing this test).

[lgritz]

I added the read test for scanline files and updated the chart above to be complete.

[lji-ilm]

Thanks Larry for the experiment!
hmm, still the single thread is much faster than exrmetrics reported.
In the exrmetrics experiment the read on 8x8 is 2s + (2000 ms +)
8x8 tile is a special case because this is the JPEG's hard-coded DCT size, which is sort of JPEG's tile or "unit" in image formation.

[lji-ilm]

Some ideas we discussed during committee meeting today:

1. exrmetrics is using the C++ warp-around-Core layer to do the read, while OIIO directly use Core. It is interesting to examine if the C++ warp-around-core layer incurs time cost for small tiles. This comparison can be done in two ways:

1.a. Compare C++ Warp-around-Core against full C++ implementation of file reads (aka EXR 3.0 and before)
1.b. Compare C++ Warp-around-Core aganist direct Core file read (we probably already have this, per Larry's experiment)

2. exrmetrics 's re-read time is not that slow. The re-read time are collected from reading a virtual exr file in the memory. So the slowdown maybe inside the OS call pattern exrmetrics issues towards a real file on disk. However, OIIO doesn't seems to have this issue and we should look at what's the difference between the OS disk call pattern of exrmetrics (which uses C++Wrap-around-Core) and OIIO (which manipulates Core directly in its EXR driver code)