Skip to content

Commit 80a802d

Browse files
jhughes-mwjhughes-mw
authored
Merge pull request #132 from mathworks/test-cleanup
Test cleanup: fixtures, coverage, and stub removal
2 parents c77e58d + b61ab3c commit 80a802d

10 files changed

Lines changed: 272 additions & 66 deletions

File tree

‎.gitignore‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,13 @@
11
# Folder for generating test Zarr files
2-
test_files/
2+
test_files/
3+
4+
# Coverage artifacts produced by tools/run_coverage.m
5+
test/coverageReport/
6+
test/cobertura.xml
7+
test/coverage_results.mat
8+
9+
# Leaked by tZarrRead/tooBigArray (runs outside a WorkingFolderFixture)
10+
test/bigData/
311

412
# Windows default autosave extension
513
*.asv
@@ -10,3 +18,8 @@ test_files/
1018
# Bytecode-compiled version of python code
1119
PythonModule/__pycache__
1220
*.pyc
21+
.venv-zarr/
22+
23+
# Local Claude Code profile (per-developer, not shared)
24+
.claude/
25+
.claude-profiles

‎CLAUDE.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Overview
6+
7+
MATLAB interface for reading and writing Zarr v2 arrays and metadata, from both local storage and Amazon S3. The MATLAB layer delegates the actual Zarr I/O to Google's [tensorstore](https://github.com/google/tensorstore) Python library, which it calls through MATLAB's `py.` Python bridge.
8+
9+
## Architecture
10+
11+
The codebase is a three-language stack. Data and type information flow across all three layers, so a change to the data path usually touches each:
12+
13+
1. **User-facing MATLAB functions** (`zarrread.m`, `zarrwrite.m`, `zarrcreate.m`, `zarrinfo.m`, `zarrwriteatt.m`) — thin wrappers that do `arguments`-block input validation and construct a `Zarr` object. These are the documented public API.
14+
15+
2. **`Zarr.m`** — the central gateway class (`classdef Zarr < handle`). It owns the connection between MATLAB and Python: bootstrapping the Python module path (`pySetup`/`ZarrPy`), building the tensorstore KVStore schema (local `file` driver vs. S3 `s3` driver), resolving/creating paths and Zarr groups, validating partial-read parameters, and converting between MATLAB and numpy arrays. Most non-trivial logic lives here as static helper methods.
16+
17+
3. **`PythonModule/ZarrPy.py`** — a small wrapper over tensorstore. Exposes `createKVStore`, `createZarr`, `writeZarr`, `readZarr`. This is the only code that talks to tensorstore directly. `Zarr.m` imports it via `py.importlib.import_module('ZarrPy')` after inserting `PythonModule/` onto `py.sys.path`.
18+
19+
### Key cross-cutting concerns
20+
21+
- **Datatype mapping** (`ZarrDatatype.m`): a single class holds three parallel arrays mapping MATLAB types ↔ tensorstore types ↔ Zarr dtype strings (e.g. `"double"` ↔ `"float64"` ↔ `"<f8"`). Construct via the static `fromMATLABType` / `fromTensorstoreType` / `fromZarrType` methods, never the private constructor. Any new supported datatype must be added to all three arrays in lockstep.
22+
23+
- **Index convention conversion**: MATLAB is 1-based and uses *count*; tensorstore is 0-based and uses *end index* (exclusive). The translation happens in `Zarr.read` (`start = start - 1`, `endInds = start + stride.*count`). Partial-read validation (Start/Stride/Count bounds, scalar-into-vector indexing) is in `Zarr.processPartialReadParams`.
24+
25+
- **Local vs. remote (S3)**: `obj.isRemote` is detected from an IRI prefix on the path. S3 URLs/URIs in six different formats are parsed into bucket + object path by `Zarr.extractS3BucketNameAndPath`. Some validity checks (e.g. `isZarrArray`) are skipped for `http`-style remote paths because they would fail even on valid arrays.
26+
27+
- **Zarr metadata files**: `.zarray` marks an array, `.zgroup` marks a group, `.zattrs` holds user-defined attributes (all Zarr v2, read/written as JSON). `zarr.json` is the Zarr v3 metadata file — it is detected by `zarrinfo` but writing v3 is not supported. `zarrinfo.m` reads these JSON files directly in MATLAB (not via Python); creating group hierarchies writes `.zgroup` files directly too.
28+
29+
## Commands
30+
31+
There is no build step — it's interpreted MATLAB plus a Python module on the path.
32+
33+
**Run the full test suite** (from the `test/` directory, since tests resolve data paths relative to `pwd`):
34+
```matlab
35+
cd test
36+
results = runtests('IncludeSubfolders', true)
37+
```
38+
39+
**Run a single test class or method:**
40+
```matlab
41+
cd test
42+
runtests('tZarrRead') % one class
43+
runtests('tZarrRead/verifyPartialArrayData') % one method
44+
```
45+
46+
CI (`.github/workflows/test_setup.yml`) runs `matlab-actions/run-tests` with `select-by-folder: 'test'` across Ubuntu/Windows/macOS and MATLAB R2024a + latest.
47+
48+
## Setup requirements
49+
50+
- MATLAB R2024a or newer. Add the repo root to the MATLAB path (`addpath`).
51+
- Python 3.10+ configured for MATLAB (`pyenv`), with `numpy` and `tensorstore` installed (see `PythonModule/requirements.txt`; CI pins `tensorstore==0.1.71`, the minimum supported version).
52+
53+
When iterating on `ZarrPy.py`, MATLAB caches the imported module. Reload with `Zarr.pyReloadInProcess()` (after `clear classes`) for in-process Python, or `terminate(pyenv)` for out-of-process.
54+
55+
## Tests
56+
57+
xUnit-style classes (`matlab.unittest.TestCase`) named `t<Feature>.m` in `test/`. They inherit shared fixtures from `SharedZarrTestSetup.m`, which adds the parent source folder to the path and copies `test/dataFiles/` into a `WorkingFolderFixture` so write tests don't pollute the repo. Read-test fixtures live in `test/dataFiles/grp_v2` (and `grp_v3`); expected results are stored in `expZarrArrData.mat` / `expZarrArrInfo.mat`.
58+
59+
## Conventions
60+
61+
- All error messages use `error("MATLAB:<area>:<id>", ...)` identifiers — match the existing namespacing when adding new ones.
62+
- Function help text is the block comment directly under the signature; keep it current since the README points users to `help <function>`.
63+
- Spelling is checked in CI by codespell (`.codespellrc`); add false positives to `ignore-words-list`.

‎codecov.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,5 @@ coverage:
55
target: 85%
66
threshold: 5%
77
ignore:
8-
- "test/*"
8+
- "test/*"
9+
- "tools/*"

‎test/tZarr.m‎

Lines changed: 94 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,99 @@ function verifyReload(testcase)
3535

3636
end
3737

38-
38+
function verifyIsZarrArrayAndGroup(testcase)
39+
% Verify that isZarrArray and isZarrGroup correctly identify a
40+
% Zarr array (has .zarray) versus a Zarr group (has .zgroup).
41+
% SharedZarrTestSetup copies the *contents* of dataFiles into
42+
% the working folder, so fixtures live at grp_v2/... directly.
43+
arrPath = "grp_v2/arr_v2";
44+
grpPath = "grp_v2";
45+
46+
testcase.verifyTrue(Zarr.isZarrArray(arrPath),...
47+
"Expected an array path to be a Zarr array.");
48+
testcase.verifyFalse(Zarr.isZarrArray(grpPath),...
49+
"Did not expect a group path to be a Zarr array.");
50+
51+
testcase.verifyTrue(Zarr.isZarrGroup(grpPath),...
52+
"Expected a group path to be a Zarr group.");
53+
testcase.verifyFalse(Zarr.isZarrGroup(arrPath),...
54+
"Did not expect an array path to be a Zarr group.");
55+
end
56+
57+
function verifyDatatypeRoundTrip(testcase)
58+
% Verify that ZarrDatatype maps consistently across MATLAB,
59+
% Tensorstore, and Zarr type names, regardless of which static
60+
% constructor is used to create it.
61+
mlType = "double";
62+
tsType = "float64";
63+
zType = "<f8";
64+
65+
fromML = ZarrDatatype.fromMATLABType(mlType);
66+
fromTS = ZarrDatatype.fromTensorstoreType(tsType);
67+
fromZarr = ZarrDatatype.fromZarrType(zType);
68+
69+
for dt = [fromML, fromTS, fromZarr]
70+
testcase.verifyEqual(dt.MATLABType, mlType);
71+
testcase.verifyEqual(dt.TensorstoreType, tsType);
72+
testcase.verifyEqual(dt.ZarrType, zType);
73+
end
74+
end
75+
76+
function verifyInvalidTensorstoreType(testcase)
77+
% Verify error when an unsupported Tensorstore type name is used.
78+
testcase.verifyError(...
79+
@()ZarrDatatype.fromTensorstoreType("not_a_type"),...
80+
"MATLAB:validators:mustBeMember");
81+
end
82+
83+
function verifyCreateGroupMakesFolder(testcase)
84+
% Verify that createGroup creates the directory and writes a
85+
% .zgroup file into it
86+
import matlab.unittest.fixtures.TemporaryFolderFixture
87+
tempFixture = testcase.applyFixture(TemporaryFolderFixture);
88+
groupPath = fullfile(tempFixture.Folder, "brandNewGroup");
89+
90+
Zarr.createGroup(groupPath);
91+
testcase.verifyTrue(isfile(fullfile(groupPath, ".zgroup")),...
92+
"createGroup should have written a .zgroup file.");
93+
end
94+
95+
function verifyCreateGroupOpenFailure(testcase)
96+
% Verify error when the .zgroup file cannot be opened for
97+
% writing. Everything lives inside an isolated temporary folder
98+
% fixture, so no real data is modified.
99+
%
100+
% We make the existing .zgroup *file* read-only rather than its
101+
% folder: a read-only folder does not prevent file creation on
102+
% Windows (the directory read-only attribute is ignored there),
103+
% whereas a read-only file is honored on both Windows and Unix.
104+
import matlab.unittest.fixtures.TemporaryFolderFixture
105+
tempFixture = testcase.applyFixture(TemporaryFolderFixture);
106+
107+
groupPath = fullfile(tempFixture.Folder, "readOnlyGroup");
108+
Zarr.createGroup(groupPath); % writes .zgroup
109+
zgroupFile = fullfile(groupPath, ".zgroup");
110+
111+
fileattrib(zgroupFile, '-w');
112+
% Restore write permission before the fixture is torn down so its
113+
% contents can be removed (runs before the fixture's rmdir).
114+
testcase.addTeardown(@()fileattrib(zgroupFile, '+w'));
115+
116+
testcase.verifyError(@()Zarr.createGroup(groupPath),...
117+
"MATLAB:Zarr:fileOpenFailure");
118+
end
119+
120+
function verifyWriteScalarShapedArray(testcase)
121+
% Verify writing to an array whose stored shape is a true scalar
122+
% (shape [1]). This exercises the isscalar(info.shape) branch of
123+
% Zarr.write, which zarrcreate cannot produce on its own because
124+
% it expands scalar sizes to [1 N].
125+
scalarPath = "grp_v2/scalarData";
126+
127+
zarrwrite(scalarPath, 42);
128+
testcase.verifyEqual(zarrread(scalarPath), 42,...
129+
"Failed to write/read a scalar-shaped Zarr array.");
130+
end
131+
39132
end
40133
end

‎test/tZarrAttributes.m‎

Lines changed: 23 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -110,14 +110,31 @@ function notZarrObject(testcase)
110110
end
111111

112112
function noWritePermissions(testcase)
113-
% Verify error if there are no write permissions to the Zarr array.
114-
115-
% Make the folder read-only.
116-
fileattrib(testcase.ArrPathWrite,'-w','','s');
117-
testcase.addTeardown(@()fileattrib(testcase.ArrPathWrite,'+w','','s'));
113+
% Verify error if the .zattrs file cannot be opened for writing.
114+
% Everything lives inside an isolated temporary folder fixture so
115+
% no shared fixture data is modified.
116+
%
117+
% We make the existing .zattrs *file* read-only rather than its
118+
% folder: a read-only folder does not prevent file creation on
119+
% Windows (the directory read-only attribute is ignored there),
120+
% whereas a read-only file is honored on both Windows and Unix.
121+
import matlab.unittest.fixtures.TemporaryFolderFixture
122+
tempFixture = testcase.applyFixture(TemporaryFolderFixture);
123+
124+
arrPath = fullfile(tempFixture.Folder, "roArr");
125+
zarrcreate(arrPath, testcase.ArrSize);
126+
% Write one attribute so the .zattrs file exists, then make it
127+
% read-only so the next write cannot open it.
128+
zarrwriteatt(arrPath, 'existingAttr', 1);
129+
zattrsFile = fullfile(arrPath, '.zattrs');
130+
131+
fileattrib(zattrsFile, '-w');
132+
% Restore write permission before the fixture is torn down so its
133+
% contents can be removed (runs before the fixture's rmdir).
134+
testcase.addTeardown(@()fileattrib(zattrsFile, '+w'));
118135

119136
errID = 'MATLAB:zarrwriteatt:fileOpenFailure';
120-
testcase.verifyError(@()zarrwriteatt(testcase.ArrPathWrite,'myAttr','attrVal'), ...
137+
testcase.verifyError(@()zarrwriteatt(arrPath,'myAttr','attrVal'), ...
121138
errID);
122139
end
123140

‎test/tZarrCreate.m‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -48,13 +48,11 @@ function createIntermediateZgroups(testcase)
4848

4949
function createArrayRelativePath(testcase)
5050
% Verify that the array is successfully created if a relative
51-
% path is used.
52-
newDir = 'myFolder';
53-
currDir = pwd;
54-
mkdir(newDir);
55-
testcase.addTeardown(@()cd(currDir));
51+
% path is used. Work from a fresh temporary folder (which the
52+
% fixture enters and cleans up) so the "../" path resolves.
53+
import matlab.unittest.fixtures.WorkingFolderFixture
54+
testcase.applyFixture(WorkingFolderFixture);
5655

57-
cd(newDir);
5856
inpPath = fullfile('..','myGrp','myArr');
5957
zarrcreate(inpPath,[10 10]);
6058
arrInfo = zarrinfo(inpPath);

‎test/tZarrInfo.m‎

Lines changed: 15 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,18 @@
1-
classdef tZarrInfo < matlab.unittest.TestCase
1+
classdef tZarrInfo < SharedZarrTestSetup
22
% Tests for zarrinfo function to get info of the Zarr file in MATLAB.
33

44
% Copyright 2025 The MathWorks, Inc.
55

66
properties(Constant)
7-
GrpPathV2 = "dataFiles/grp_v2"
8-
ArrPathV2 = "dataFiles/grp_v2/arr_v2"
9-
GrpPathV3 = "dataFiles/grp_v3"
10-
ArrPathV3 = "dataFiles/grp_v3/arr_v3"
11-
ExpInfo = load(fullfile(pwd,"dataFiles","expZarrArrInfo.mat"))
12-
end
7+
% SharedZarrTestSetup copies the contents of dataFiles/ into the
8+
% working folder, so fixtures are at the working folder root.
9+
GrpPathV2 = "grp_v2"
10+
ArrPathV2 = "grp_v2/arr_v2"
11+
GrpPathV3 = "grp_v3"
12+
ArrPathV3 = "grp_v3/arr_v3"
1313

14-
methods(TestClassSetup)
15-
function addSrcCodePath(testcase)
16-
% Add source code path before running the tests
17-
import matlab.unittest.fixtures.PathFixture
18-
testcase.applyFixture(PathFixture(fullfile('..'),'IncludeSubfolders',true))
19-
end
14+
% Loaded at class-load time, while pwd is still the test folder.
15+
ExpInfo = load(fullfile(pwd,"dataFiles","expZarrArrInfo.mat"))
2016
end
2117

2218
methods(Test)
@@ -35,9 +31,12 @@ function verifyGroupInfoV2(testcase)
3531
end
3632

3733
function getArrayInfoRelativePath(testcase)
38-
% Verify array info if the input is using relative path to the
39-
% array.
40-
inpPath = fullfile('..','test',testcase.ArrPathV2);
34+
% Verify array info if the input is using a relative path to the
35+
% array. Read from a subfolder using a "../" prefixed path.
36+
import matlab.unittest.fixtures.CurrentFolderFixture
37+
testcase.applyFixture(CurrentFolderFixture("grp_v2"));
38+
39+
inpPath = fullfile('..', testcase.ArrPathV2);
4140
actInfo = zarrinfo(inpPath);
4241
expInfo = testcase.ExpInfo.zarrV2ArrInfo;
4342
testcase.verifyEqual(actInfo, expInfo, ['Failed to verify array info ' ...

‎test/tZarrRead.m‎

Lines changed: 18 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,23 @@
1-
classdef tZarrRead < matlab.unittest.TestCase
1+
classdef tZarrRead < SharedZarrTestSetup
22
% Tests for zarrread function to read data from Zarr files in MATLAB.
33

44
% Copyright 2025 The MathWorks, Inc.
55

66
properties(Constant)
7-
% Path for read functions
8-
GrpPathRead = "dataFiles/grp_v2"
9-
ArrPathRead = "dataFiles/grp_v2/arr_v2"
10-
ArrPathReadSmall = "dataFiles/grp_v2/smallArr"
11-
ArrPathReadVector = "dataFiles/grp_v2/vectorData"
12-
ArrPathReadScalar = "dataFiles/grp_v2/scalarData"
13-
ArrPathReadV3 = "dataFiles/grp_v3/arr_v3"
14-
7+
% Paths for read functions. SharedZarrTestSetup copies the contents
8+
% of dataFiles/ into the working folder, so fixtures are at the
9+
% working folder root (grp_v2/..., grp_v3/...).
10+
GrpPathRead = "grp_v2"
11+
ArrPathRead = "grp_v2/arr_v2"
12+
ArrPathReadSmall = "grp_v2/smallArr"
13+
ArrPathReadVector = "grp_v2/vectorData"
14+
ArrPathReadScalar = "grp_v2/scalarData"
15+
ArrPathReadV3 = "grp_v3/arr_v3"
16+
17+
% Loaded at class-load time, while pwd is still the test folder.
1518
ExpData = load(fullfile(pwd,"dataFiles","expZarrArrData.mat"))
1619
end
1720

18-
methods(TestClassSetup)
19-
function addSrcCodePath(testcase)
20-
% Add source code path before running the tests
21-
import matlab.unittest.fixtures.PathFixture
22-
testcase.applyFixture(PathFixture(fullfile('..'),'IncludeSubfolders',true))
23-
end
24-
end
25-
2621
methods(Test)
2722
function verifyArrayData(testcase)
2823
% Verify array data using zarrread function.
@@ -88,9 +83,12 @@ function verifyReadScalarData(testcase)
8883
end
8984

9085
function verifyArrayDataRelativePath(testcase)
91-
% Verify array data if the input is using relative path to the
92-
% array.
93-
inpPath = fullfile('..','test',testcase.ArrPathRead);
86+
% Verify array data if the input is using a relative path to the
87+
% array. Read from a subfolder using a "../" prefixed path.
88+
import matlab.unittest.fixtures.CurrentFolderFixture
89+
testcase.applyFixture(CurrentFolderFixture("grp_v2"));
90+
91+
inpPath = fullfile('..', testcase.ArrPathRead);
9492
actArrData = zarrread(inpPath);
9593
expArrData = testcase.ExpData.arr_v2;
9694
testcase.verifyEqual(actArrData,expArrData,['Failed to verify array ' ...

‎test/tZarrWrite.m‎

Lines changed: 0 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -28,13 +28,6 @@ function createArrayLocalDefaultSyntax(testcase,ArrSizeWrite)
2828
testcase.verifyEqual(actData,expData,'Failed to verify array data');
2929
end
3030

31-
function createArrayRemoteDefaultSyntax(testcase)
32-
% Verify data when creating and writing to arrays of different
33-
% dimensions using zarrcreate and zarrwrite to a remote location.
34-
35-
% Move to a separate file
36-
end
37-
3831
function createArrayLocalUserDefinedSyntax(testcase,DataType,CompId)
3932
% Verify the data when creating and writing to arrays with
4033
% user-defined properties using zarrcreate and zarrwrite locally.
@@ -51,14 +44,6 @@ function createArrayLocalUserDefinedSyntax(testcase,DataType,CompId)
5144
' with ' CompId ' compression.']);
5245
end
5346

54-
function createArrayRemoteUserDefinedSyntax(testcase)
55-
% Verify data when creating and writing data to arrays with
56-
% user-defined properties using zarrcreate and zarrwrite to a
57-
% remote location.
58-
59-
% Move to a separate file
60-
end
61-
6247
function createArrayWithDefaultBloscConfig(testcase)
6348
% Verify data when creating and writing to a Zarr array using
6449
% a default blosc compression configuration.

0 commit comments

Comments
 (0)