Skip to content

Take axes metadata into account #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dominikl wants to merge 19 commits into from
Closed

Take axes metadata into account #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
63c9936
Take axes metadata into account
dominikl Apr 30, 2025
71c93bb
Add test checking default axes if not present
jburel Jun 4, 2025
dfe6319
Fix order since order in omero is reverse in ome.zarr
jburel Jun 4, 2025
38838db
Merge pull request #1 from jburel/axes
dominikl Jun 5, 2025
d8a7e3c
Remove printlns
dominikl Jun 5, 2025
7fef6bf
Remove commented out code
dominikl Jun 5, 2025
b6dc260
Remove unused imports
dominikl Jun 5, 2025
7d9f338
Support zarr without c, t or z axes
dominikl Jun 5, 2025
a7cd5c9
Fix swapped xy tile size
dominikl Jun 6, 2025
0020bb6
Fix issue when Z missing
dominikl Jun 6, 2025
e4eb3e9
Fix issue with checkReadSize
dominikl Jun 6, 2025
ed93541
Add integration tests for different dimensions
dominikl Jun 11, 2025
8995b11
remove logging from TestZarr
dominikl Jun 12, 2025
56df523
Moved classes into correct package
dominikl Jun 13, 2025
04fe46e
Fix package declaration
dominikl Jun 13, 2025
de1096e
Fix order
jburel Jun 13, 2025
51119b5
install blosc
jburel Jun 13, 2025
0a30109
Check repo owner
jburel Jun 13, 2025
d000619
Merge pull request #2 from jburel/axes
dominikl Jun 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,12 @@ jobs:
with:
java-version: ${{ matrix.java }}
distribution: 'zulu'
- name: Install dependency
run: sudo apt-get install libblosc1
- name: Run commands
run: |
./gradlew ${{ env.gradle_commands }}
- name: Publish artifacts
if: github.event_name != 'pull_request' && matrix.java == 11
if: github.event_name != 'pull_request' && matrix.java == 11 && github.repository_owner == 'glencoesoftware'
run: |
./gradlew -PArtifactoryUserName=${ArtifactoryUserName} -PArtifactoryPassword=${ArtifactoryPassword} publish
226 changes: 180 additions & 46 deletions src/main/java/com/glencoesoftware/omero/zarr/ZarrPixelBuffer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,13 @@ public class ZarrPixelBuffer implements PixelBuffer {
/** Array path vs. ZarrArray cache */
private final AsyncLoadingCache<Path, ZarrArray> zarrArrayCache;

public enum Axes {
X, Y, Z, C, T;
Copy link
Copy Markdown
Member

@sbesson sbesson Jun 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The usage of enums here is probably conditional to whether we want to handle OME-Zarr datasets with dimensions that are not one of xyzct

}

/** Maps axes to their corresponding array indexes */
private Map<Axes, Integer> axes;
Copy link
Copy Markdown
Member

@sbesson sbesson Jun 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

During review, I found the usage of axes confusing as it refers to:

  • an enum for the XYZCT dimension names
  • a map allowing to store the axes metadata


/**
* Default constructor
* @param pixels Pixels metadata for the pixel buffer
Expand Down Expand Up @@ -122,12 +129,17 @@ public ZarrPixelBuffer(Pixels pixels, Path root, Integer maxPlaneWidth,
if (!rootGroupAttributes.containsKey("multiscales")) {
throw new IllegalArgumentException("Missing multiscales metadata!");
}
this.axes = getAxes();
if (!axes.containsKey(Axes.X) || !axes.containsKey(Axes.Y)) {
throw new IllegalArgumentException("Missing X or Y axis!");
Copy link
Copy Markdown
Member

@sbesson sbesson Jun 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming additional validation is performed on the axes dimensions, should this logic happen here or directly in getAxes?

}
this.resolutionLevels = this.getResolutionLevels();
setResolutionLevel(this.resolutionLevels - 1);
if (this.resolutionLevel < 0) {
throw new IllegalArgumentException(
"This Zarr file has no pixel data");
}

this.maxPlaneWidth = maxPlaneWidth;
this.maxPlaneHeight = maxPlaneHeight;

Expand Down Expand Up @@ -196,25 +208,30 @@ private long length(int[] shape) {
private void read(byte[] buffer, int[] shape, int[] offset)
throws IOException {
// Check planar read size (sizeX and sizeY only)
checkReadSize(Arrays.copyOfRange(shape, 3, 5));

checkReadSize(new int[] {shape[axes.get(Axes.X)], shape[axes.get(Axes.Y)]});
// if reading from a resolution downsampled in Z,
// adjust the shape/offset for the Z coordinate only
// this ensures that the correct Zs are read from the correct offsets
// since the requested shape/offset may not match the underlying array
int planes = 1;
int originalZIndex = offset[2];
int originalZIndex = 1;
if (axes.containsKey(Axes.Z)) {
originalZIndex = offset[axes.get(Axes.Z)];
}
if (getSizeZ() != getTrueSizeZ()) {
offset[2] = zIndexMap.get(originalZIndex);
planes = shape[2];
shape[2] = 1;
offset[axes.get(Axes.Z)] = zIndexMap.get(originalZIndex);
planes = shape[axes.get(Axes.Z)];
shape[axes.get(Axes.Z)] = 1;
}

try {
ByteBuffer asByteBuffer = ByteBuffer.wrap(buffer);
DataType dataType = array.getDataType();
for (int z=0; z<planes; z++) {
offset[2] = zIndexMap.get(originalZIndex + z);
if (axes.containsKey(Axes.Z)) {
offset[axes.get(Axes.Z)] = zIndexMap.get(originalZIndex + z);
}
switch (dataType) {
case u1:
case i1:
Expand Down Expand Up @@ -296,7 +313,7 @@ public int[][] getChunks() throws IOException {
}

/**
* Retrieves the datasets metadat of the first multiscale from the root
* Retrieves the datasets metadata of the first multiscale from the root
* group attributes.
* @return See above.
* @see #getRootGroupAttributes()
Expand All @@ -307,6 +324,32 @@ public List<Map<String, String>> getDatasets() {
getMultiscalesMetadata().get(0).get("datasets");
}

/**
* Retrieves the axes metadata of the first multiscale
* @return See above.
*/
public Map<Axes, Integer> getAxes() {
HashMap<Axes, Integer> axes;
try {
axes = new HashMap<>();
List<Map<String, Object>> axesData = (List<Map<String, Object>>) getMultiscalesMetadata().get(0).get("axes");
for (int i=0; i<axesData.size(); i++) {
Map<String, Object> axis = axesData.get(i);
String name = axis.get("name").toString().toUpperCase();
axes.put(Axes.valueOf(name), i);
Copy link
Copy Markdown
Member

@sbesson sbesson Jun 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the expectation if axes.get(i).get("name") is not in the list of enums?

}
} catch (Exception e) {
log.warn("No axes metadata found, defaulting to standard axes TCZYX");
Copy link
Copy Markdown
Member

@sbesson sbesson Jun 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this logic only happen when there is no axes metadata?
There are a few conditions under which the block above will throw an exception and fallback to the default 5D dimension order - is that the expectations?

axes = new HashMap<>();
axes.put(Axes.T, 0);
axes.put(Axes.C, 1);
axes.put(Axes.Z, 2);
axes.put(Axes.Y, 3);
axes.put(Axes.X, 4);
}
return axes;
}

/**
* Retrieves the multiscales metadata from the root group attributes.
* @return See above.
Expand Down Expand Up @@ -517,8 +560,33 @@ public byte[] getTileDirect(
checkBounds(x, y, z, c, t);
//Check check bottom-right of tile in bounds
checkBounds(x + w - 1, y + h - 1, z, c, t);
int[] shape = new int[] { 1, 1, 1, h, w };
int[] offset = new int[] { t, c, z, y, x };

int[] shape = new int[axes.size()];
if (axes.containsKey(Axes.T)) {
shape[axes.get(Axes.T)] = 1;
Copy link
Copy Markdown
Member

@sbesson sbesson Jun 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor offset[axes.get(Axes.T)] = t; could be moved here to reduce a bit of duplication (same in in all the lines below)

}
if (axes.containsKey(Axes.C)) {
shape[axes.get(Axes.C)] = 1;
}
if (axes.containsKey(Axes.Z)) {
shape[axes.get(Axes.Z)] = 1;
}
shape[axes.get(Axes.Y)] = h;
shape[axes.get(Axes.X)] = w;

int[] offset = new int[axes.size()];
if (axes.containsKey(Axes.T)) {
offset[axes.get(Axes.T)] = t;
}
if (axes.containsKey(Axes.C)) {
offset[axes.get(Axes.C)] = c;
}
if (axes.containsKey(Axes.Z)) {
offset[axes.get(Axes.Z)] = z;
}
offset[axes.get(Axes.Y)] = y;
offset[axes.get(Axes.X)] = x;

read(buffer, shape, offset);
return buffer;
} catch (Exception e) {
Expand Down Expand Up @@ -618,8 +686,33 @@ public byte[] getStackDirect(Integer c, Integer t, byte[] buffer)
checkBounds(x, y, z, c, t);
//Check check bottom-right of tile in bounds
checkBounds(x + w - 1, y + h - 1, z, c, t);
int[] shape = new int[] { 1, 1, getSizeZ(), h, w };
int[] offset = new int[] { t, c, z, y, x };

int[] shape = new int[axes.size()];
if (axes.containsKey(Axes.T)) {
shape[axes.get(Axes.T)] = 1;
Copy link
Copy Markdown
Member

@sbesson sbesson Jun 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

}
if (axes.containsKey(Axes.C)) {
shape[axes.get(Axes.C)] = 1;
}
if (axes.containsKey(Axes.Z)) {
shape[axes.get(Axes.Z)] = getSizeZ();
}
shape[axes.get(Axes.Y)] = h;
shape[axes.get(Axes.X)] = w;

int[] offset = new int[axes.size()];
if (axes.containsKey(Axes.T)) {
offset[axes.get(Axes.T)] = t;
}
if (axes.containsKey(Axes.C)) {
offset[axes.get(Axes.C)] = c;
}
if (axes.containsKey(Axes.Z)) {
offset[axes.get(Axes.Z)] = z;
}
offset[axes.get(Axes.Y)] = y;
offset[axes.get(Axes.X)] = x;

read(buffer, shape, offset);
return buffer;
}
Expand Down Expand Up @@ -647,8 +740,33 @@ public byte[] getTimepointDirect(Integer t, byte[] buffer)
checkBounds(x, y, z, c, t);
//Check check bottom-right of tile in bounds
checkBounds(x + w - 1, y + h - 1, z, c, t);
int[] shape = new int[] { 1, getSizeC(), getSizeZ(), h, w };
int[] offset = new int[] { t, c, z, y, x };

int[] shape = new int[axes.size()];
if (axes.containsKey(Axes.T)) {
shape[axes.get(Axes.T)] = 1;
Copy link
Copy Markdown
Member

@sbesson sbesson Jun 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

}
if (axes.containsKey(Axes.C)) {
shape[axes.get(Axes.C)] = getSizeC();
}
if (axes.containsKey(Axes.Z)) {
shape[axes.get(Axes.Z)] = getSizeZ();
}
shape[axes.get(Axes.Y)] = h;
shape[axes.get(Axes.X)] = w;

int[] offset = new int[axes.size()];
if (axes.containsKey(Axes.T)) {
offset[axes.get(Axes.T)] = t;
}
if (axes.containsKey(Axes.C)) {
offset[axes.get(Axes.C)] = c;
}
if (axes.containsKey(Axes.Z)) {
offset[axes.get(Axes.Z)] = z;
}
offset[axes.get(Axes.Y)] = y;
offset[axes.get(Axes.X)] = x;

read(buffer, shape, offset);
return buffer;
}
Expand Down Expand Up @@ -757,35 +875,47 @@ public long getId() {

@Override
public int getSizeX() {
return array.getShape()[4];
return array.getShape()[axes.get(Axes.X)];
}

@Override
public int getSizeY() {
return array.getShape()[3];
return array.getShape()[axes.get(Axes.Y)];
}

@Override
public int getSizeZ() {
// this is expected to be the Z size of the full resolution array
return zIndexMap.size();
if (axes.containsKey(Axes.Z)) {
// this is expected to be the Z size of the full resolution array
return zIndexMap.size();
}
return 1;
}

/**
* @return Z size of the current underlying Zarr array
*/
private int getTrueSizeZ() {
return array.getShape()[2];
if (axes.containsKey(Axes.Z)) {
return array.getShape()[axes.get(Axes.Z)];
}
return 1;
}

@Override
public int getSizeC() {
return array.getShape()[1];
if (axes.containsKey(Axes.C)) {
return array.getShape()[axes.get(Axes.C)];
}
return 1;
}

@Override
public int getSizeT() {
return array.getShape()[0];
if (axes.containsKey(Axes.T)) {
return array.getShape()[axes.get(Axes.T)];
}
return 1;
}

@Override
Expand Down Expand Up @@ -813,38 +943,42 @@ public void setResolutionLevel(int resolutionLevel) {
throw new IllegalArgumentException(
"This Zarr file has no pixel data");
}
if (zIndexMap == null) {
zIndexMap = new HashMap<Integer, Integer>();
}
else {
zIndexMap.clear();
}
try {
array = zarrArrayCache.get(
root.resolve(Integer.toString(this.resolutionLevel))).get();

ZarrArray fullResolutionArray = zarrArrayCache.get(
root.resolve("0")).get();

// map each Z index in the full resolution array
// to a Z index in the subresolution array
// if no Z downsampling, this is just an identity map
int fullResZ = fullResolutionArray.getShape()[2];
int arrayZ = array.getShape()[2];
for (int z=0; z<fullResZ; z++) {
zIndexMap.put(z, Math.round(z * arrayZ / fullResZ));

if (zIndexMap == null) {
Copy link
Copy Markdown
Member

@sbesson sbesson Jun 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the extra indent on purpose here and in the following lines?

zIndexMap = new HashMap<Integer, Integer>();
}
} catch (Exception e) {
// FIXME: Throw the right exception
throw new RuntimeException(e);
}
else {
zIndexMap.clear();
}
try {
array = zarrArrayCache.get(
root.resolve(Integer.toString(this.resolutionLevel))).get();

ZarrArray fullResolutionArray = zarrArrayCache.get(
root.resolve("0")).get();

if (axes.containsKey(Axes.Z)) {
// map each Z index in the full resolution array
// to a Z index in the subresolution array
// if no Z downsampling, this is just an identity map
int fullResZ = fullResolutionArray.getShape()[axes.get(Axes.Z)];
int arrayZ = array.getShape()[axes.get(Axes.Z)];
for (int z=0; z<fullResZ; z++) {
zIndexMap.put(z, Math.round(z * arrayZ / fullResZ));
}
}
} catch (Exception e) {
// FIXME: Throw the right exception
throw new RuntimeException(e);
}

}

@Override
public Dimension getTileSize() {
try {
int[] chunks = getChunks()[resolutionLevel];
return new Dimension(chunks[4], chunks[3]);
return new Dimension(chunks[axes.get(Axes.X)], chunks[axes.get(Axes.Y)]);
} catch (Exception e) {
// FIXME: Throw the right exception
throw new RuntimeException(e);
Expand Down
Loading