Testbed tool, conformance report table and reference datasets

[xulman]

This PR is currently a work-in-progress.

Its purpose is to have two entry test methods

• A pure unit-test method that opens a given dataset using a given backend, shows an IJ window (to simulate a real user wanting to display an OME-Zarr dataset), and checks various parameters with asserts. This method is called iteratively over a collection of test data produced by testCasesFeeder().
• Another method that iteratively calls the previous one, monitors its asserts, and builds up a conformance tests overview table, see below.

The first method is clearly the workhorse here. Additionally, the class TestDataset is crucial to the method's testing capability, as it contains a pointer to the source dataset and numerous dataset parameters, such as axis sizes, the number of resolution levels, pixel resolutions, axis order, etc. The workhorse method only verifies matches between the actual values (from the PyramidDataset object created at runtime) and the reference values (from the TestDataset object) over the said parameters.

The folder /testing currently holds aux tools to create a .csv table, see also #88, that feeds and arrives into TestDataset objects.

The second method reportSuccessRateAndPrintTable() can give an output like this:

+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
| Dataset                                                                        | Backend   | Result | Reason                                                                         |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                            file://...../2d_dataset_v5.ome.zarr (idr0106) v0.5: |        N5 |   PASS |                                                                                |
|                         size=(64 x 64 x 1 x 1 x 1) [XYZCT], axes='XY', pxType= |           |        |                                                                                |
|     multiscales=-1, wells=-1, fields=-1, resolution=(-1.0 x -1.0 x -1.0) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                            file://...../2d_dataset_v5.ome.zarr (idr0106) v0.5: | ZARR_JAVA |   PASS |                                                                                |
|                         size=(64 x 64 x 1 x 1 x 1) [XYZCT], axes='XY', pxType= |           |        |                                                                                |
|     multiscales=-1, wells=-1, fields=-1, resolution=(-1.0 x -1.0 x -1.0) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                 https://ome-z.....s.com/...../miranda.ome.zarr (miranda) v0.5: |        N5 |   PASS |                                                                                |
|        size=(1024 x 1024 x 1024 x -1 x -1) [XYZCT], axes='XYZ', pxType=float32 |           |        |                                                                                |
|         multiscales=4, wells=-1, fields=-1, resolution=(1.0 x 1.0 x 1.0) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                 https://ome-z.....s.com/...../miranda.ome.zarr (miranda) v0.5: | ZARR_JAVA |   FAIL | AssertionFailedError: Failed opening dataset 'miranda' with IJ. ==> Unexpected |
|        size=(1024 x 1024 x 1024 x -1 x -1) [XYZCT], axes='XYZ', pxType=float32 |           |        |                                  exception thrown: java.lang.RuntimeException: |
|         multiscales=4, wells=-1, fields=-1, resolution=(1.0 x 1.0 x 1.0) [XYZ] |           |        |                                       java.util.concurrent.ExecutionException: |
|                                                                                |           |        |                                       java.lang.ArrayIndexOutOfBoundsException |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|     https://ome-z.....s.com/...../marmoset_neurons.ome.zarr (marmoset_neurons) |        N5 |   PASS |                                                                                |
|                                                                          v0.5: |           |        |                                                                                |
|           size=(1024 x 1024 x 314 x -1 x -1) [XYZCT], axes='XYZ', pxType=uint8 |           |        |                                                                                |
|     multiscales=4, wells=-1, fields=-1, resolution=(0.497 x 0.497 x 1.5) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|     https://ome-z.....s.com/...../marmoset_neurons.ome.zarr (marmoset_neurons) | ZARR_JAVA |   PASS |                                                                                |
|                                                                          v0.5: |           |        |                                                                                |
|           size=(1024 x 1024 x 314 x -1 x -1) [XYZCT], axes='XYZ', pxType=uint8 |           |        |                                                                                |
|     multiscales=4, wells=-1, fields=-1, resolution=(0.497 x 0.497 x 1.5) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                     https://livin.....ac.uk/...../9798462.zarr (idr0073) v0.4: |        N5 |   PASS |                                                                                |
|                size=(21115 x 16433 x 1 x 3 x 1) [XYZCT], axes='XYZCT', pxType= |           |        |                                                                                |
|     multiscales=-1, wells=-1, fields=-1, resolution=(-1.0 x -1.0 x -1.0) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                     https://livin.....ac.uk/...../9798462.zarr (idr0073) v0.4: | ZARR_JAVA |   FAIL | AssertionFailedError: Failed opening dataset 'idr0073' with IJ. ==> Unexpected |
|                size=(21115 x 16433 x 1 x 3 x 1) [XYZCT], axes='XYZCT', pxType= |           |        |                                                              exception thrown: |
|     multiscales=-1, wells=-1, fields=-1, resolution=(-1.0 x -1.0 x -1.0) [XYZ] |           |        |   sc.fiji.ome.zarr.pyramid.exceptions.PyramidLevelAccessException: Cannot open |
|                                                                                |           |        |                                                         resolution level 0 of: |
|                                                                                |           |        |            https://livingobjects.ebi.ac.uk/idr/zarr/v0.4/idr0073A/9798462.zarr |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                        https://livin.....ac.uk/...../9512.zarr (idr0072) v0.4: |        N5 |   FAIL |      AssertionFailedError: After opening, no Dataset is available in Fiji. ==> |
|                    size=(1345 x 1017 x 1 x 2 x 1) [XYZCT], axes='XYC', pxType= |           |        |                                                     expected: <1> but was: <0> |
|     multiscales=-1, wells=72, fields=20, resolution=(-1.0 x -1.0 x -1.0) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+
|                        https://livin.....ac.uk/...../9512.zarr (idr0072) v0.4: | ZARR_JAVA |   FAIL |      AssertionFailedError: After opening, no Dataset is available in Fiji. ==> |
|                    size=(1345 x 1017 x 1 x 2 x 1) [XYZCT], axes='XYC', pxType= |           |        |                                                     expected: <1> but was: <0> |
|     multiscales=-1, wells=72, fields=20, resolution=(-1.0 x -1.0 x -1.0) [XYZ] |           |        |                                                                                |
+--------------------------------------------------------------------------------+-----------+--------+--------------------------------------------------------------------------------+

Passed: 6 / 10

TODO

• Fully populate the source .csv table, as it is currently missing values for some datasets. These are the '-1' or '' values.
• Consider creating and including dummy OME-Zarr datasets that would be created "around" .json files from the official ngff spec repo.
• Consider using our internal dataset (in the resources folder).
• Move the .csv to the resources section of the package.

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud