chore: Update output data validations for frames and ctf #484

Bento007 · 2025-04-16T23:24:44Z

Description

resolves chanzuckerberg/cryoet-data-portal#1528

Validations for MDOC/Frames

number of mdoc sections == number of frame files on s3 == number of items in frames table/metadata
frame file names match file names in mdoc section
tilt axis angle in mdoc file matches that in tilt series metadata (+/- 10 deg)
tilt axis angle in mdoc file matches that in the alignment metadata [PSAP. in_plane_rotation] (+/- 10 deg)
number of frames >= # of per section parameters

Validations for Frames Metadata

When isGainCorrected == False, a Gains entity exists for the run
max(acquisitionOrder) <= number of frames - 1
Sum of exposureDose of all frames associated with a tilt series == totalFlux+-1 of tilt series
Sorting acquisitionOrder low-to-high and accumulatedDose low-to-high results in the same order

Validations for PerSectionParameter Metadata

-180 <= astigmaticAngle <= 180
majorDefocus > minorDefocus
0 <= phaseShift <= 2*pi
maxResolution > 0
rawAngle matches mdoc TiltAngle (+-10^-3 deg).
0 <= zIndex <= (z-Dimension of tilt series - 1)

Extra

generalize helper_angles_injection_errors angle_tolerance to work with more use cases.
Remove len(errors) == 0 assertion when multiple errors are caught. This reduces noise in the error message received.
Add frame_metadata and frames_meta_file pytest fixtures to facilitate testing.

- update poetry lock

…ose low-to-high results in the same order" - add test for "Sum of exposureDose of all frames associated with a tilt series == totalFlux of tilt series" - add test for "max(acquisitionOrder) < number of frames - 1" - add test for "When isGainCorrected == False, a Gains entity exists for the run" - test_mdoc_frame_paths covers "frame file names match file names in mdoc section" - updated test_mdoc_frames to cover "number of mdoc sections == number of frame files on s3 == number of items in frames table/metadata" - There is no per_section_alignment_parameter.rotation_angle so I'm using tilt_angle. - test_tilt_axis_angle - add test for "number of frames >= # of per section parameters"

- -180 <= astigmaticAngle <= 180 - majorDefocus > minorDefocus - 0 <= phaseShift <= 2*pi - maxResolution > 0 - rawAngle matches mdoc angle as defined above - 0 <= zIndex <= (z-Dimension of tilt series - 1)

Bento007 · 2025-04-17T20:54:48Z

tilt axis angle in mdoc file matches that in the alignment metadata [PSAP.rotation_angle] (+/- 10 deg)

@uermel I don't see rotation_angle in Per Section Alignment Parameter(PSAP.rotation_angle). What value in PSAP should be compared to tiltseries_tilt_axis

Bento007 · 2025-04-17T20:55:44Z

max(acquisitionOrder) < number of frames - 1

@uermel Should this be "max(acquisitionOrder) <= number of frames - 1."

uermel · 2025-04-18T20:01:58Z

tilt axis angle in mdoc file matches that in the alignment metadata [PSAP.rotation_angle] (+/- 10 deg)

@uermel I don't see rotation_angle in Per Section Alignment Parameter(PSAP.rotation_angle). What value in PSAP should be compared to tiltseries_tilt_axis

@Bento007 It should be in_plane_rotation.

uermel · 2025-04-18T20:03:46Z

max(acquisitionOrder) < number of frames - 1

@uermel Should this be "max(acquisitionOrder) <= number of frames - 1."

@Bento007 Yes, that's right.

…ection_alignment_parameters

Bento007 · 2025-04-18T22:22:07Z

testing with the following datasets 10014, 10447, 10085, 10156

10014

FAILED: "Tiltseries: sum of exposureDose of all frames associated with a tilt series == totalFlux of tilt series"
tests/test_tiltseries.py:130 (TestTiltseries.test_exposure_dose[10014-pda2021-01-12-1-100])
45.001599999999996 != 50.0

Expected :50.0
Actual :45.001599999999996

uermel · 2025-04-18T22:56:20Z

testing with the following datasets 10014, 10447, 10085, 10156

10014

FAILED: "Tiltseries: sum of exposureDose of all frames associated with a tilt series == totalFlux of tilt series" tests/test_tiltseries.py:130 (TestTiltseries.test_exposure_dose[10014-pda2021-01-12-1-100]) 45.001599999999996 != 50.0

Expected :50.0 Actual :45.001599999999996

@Bento007 Interesting! The validations already find mistakes! The total flux should indeed be 45 for this (based on the methods in the associated publication), I need to fix that in the config. Forgot to update total flux and only updated dose rate.

Can we please compare this with a pytest.approx with a precision of ~1? Due to the precision of the values for dose rate this may never match exactly.

Copilot

Pull Request Overview

This PR updates and expands the data validations for MDOC/Frames, Frames Metadata, and PerSectionParameter Metadata while adding additional pytest fixtures to support testing.

Adds new tests for tiltseries and frame validations including exposure dose, per section parameters, tilt axis angle comparisons, etc.
Updates error handling in tests to raise AssertionErrors instead of simple assertions.
Enhances fixtures and helper functions to support new file filtering and metadata loading.

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
ingestion_tools/scripts/data_validation/standardized/tests/test_tiltseries.py	Adds tests for exposure dose, per section parameter count, astigmatic angle, phase shift, max resolution, raw angle, and z_index validations.
ingestion_tools/scripts/data_validation/standardized/tests/test_frame.py	Introduces tests for gain correction, acquisition order, and sorting of frames.
ingestion_tools/scripts/data_validation/standardized/tests/test_alignments.py	Implements tests for tilt angle consistency between mdoc and alignment metadata with improved error handling.
ingestion_tools/scripts/data_validation/standardized/fixtures/path.py	Updates file filtering to exclude .json files in addition to .mdoc.
ingestion_tools/scripts/data_validation/standardized/fixtures/data.py	Adds a fixture to load frame metadata from frames_metadata.json.
ingestion_tools/scripts/data_validation/shared/helper/tiltseries_helper.py	Updates tilt axis angle test to verify consistency with a 10° tolerance.
ingestion_tools/scripts/data_validation/shared/helper/tilt_angles_helper.py	Adjusts error raising for raw tilt tests.
ingestion_tools/scripts/data_validation/shared/helper/mdoc_helper.py	Enhances mdoc tests to include verification of frame metadata length and more robust error reporting.
ingestion_tools/scripts/data_validation/shared/helper/angles_helper.py	Updates helper to utilize the passed angle_tolerance parameter.

ingestion_tools/scripts/data_validation/standardized/tests/test_tiltseries.py

ingestion_tools/scripts/data_validation/standardized/tests/test_frame.py

ingestion_tools/scripts/data_validation/standardized/tests/test_alignments.py

Co-authored-by: Copilot <[email protected]>

Bento007 · 2025-04-21T23:21:06Z

A couple of the test datasets are failing but they look like actual errors with the data and not the tests

manasaV3 · 2025-04-22T16:33:19Z

ingestion_tools/scripts/data_validation/standardized/fixtures/path.py

+    if filesystem.exists(dst):
+        return dst
+    else:
+        pytest.fail(f"Frames metadata file not found: {dst}")


Could we update the message such that, it informs that Frames directory exists but the metadata.json doesn't?

manasaV3 · 2025-04-22T16:37:52Z

ingestion_tools/scripts/data_validation/standardized/tests/test_alignments.py

+            pytest.skip("Alignment metadata missing per_section_alignment_parameters.")
+        errors = []
+        for i, psap in enumerate(per_section_alignment_parameters):
+            if (in_plane_rotation := psap.get("in_plane_rotation")) is not None:


iirc, the in_plane_rotation is a 2x2 matrix. should there be a conversion for it before the following check?

Is there a specific way we do that conversion?

manasaV3 · 2025-04-22T16:42:31Z

ingestion_tools/scripts/data_validation/standardized/tests/test_frame.py

+                         frame_metadata: Dict,
+                         gain_headers: Dict[str, Union[List[tifffile.TiffPage], MrcInterpreter]], # this is skipped if it is not found
+                         ):
+        if frame_metadata.get("is_gain_corrected") is False: # todo is none considered false?


I would treat None as False for this case.

By that you mean the if condition should be true, if is_gain_corrected is None, which would be equivalent to

Suggested change

if frame_metadata.get("is_gain_corrected") is False: # todo is none considered false?

if not frame_metadata.get("is_gain_corrected"):

manasaV3 · 2025-04-22T16:43:02Z

ingestion_tools/scripts/data_validation/standardized/tests/test_frame.py

+    @allure.title("Frames: max(acquisitionOrder) <= number of frames -1")
+    def test_max_acquisition_order(self, frame_metadata: Dict):
+        acquisition_order_max = max(f.get("acquisition_order", 0) for f in frame_metadata["frames"])
+        assert acquisition_order_max <= len(frame_metadata["frames"]) -1


nit: space after - ?

manasaV3 · 2025-04-22T16:45:16Z

ingestion_tools/scripts/data_validation/standardized/tests/test_tiltseries.py

+    def test_exposure_dose(self, frame_metadata: Dict, tiltseries_metadata: Dict):
+        assert sum(f.get("exposure_dose", 0) for f in frame_metadata["frames"]) == pytest.approx(tiltseries_metadata["total_flux"], abs=1)
+
+    @allure.title("PerSectionParameters: number of frames >= # of per section parameters.")


The label of what it is testing against like "PerSectionParameters" is very neat! 😄

ingestion_tools/scripts/data_validation/standardized/tests/test_tiltseries.py

- fixture frames_meta_file failure message indicate the frames directory exists. - frame_metadata["is_gain_corrected"] is None is treated as - move mdoc_tilt_axis_angle outside TiltSeriesHelper to make it easier to share. - Add function to convert matrix to degress to be used in test_mdoc_tilt_axis_angle_in_alignment_per_section_alignment_parameters Signed-off-by: Bento007 <[email protected]>

Signed-off-by: Bento007 <[email protected]>

Bento007 added 4 commits April 11, 2025 13:12

- add frame_metadata fixture

b482456

- update poetry lock

change to "max(acquisitionOrder) < number of frames"

2887959

Validations for PerSectionParameter Metadata

e3d8171

- -180 <= astigmaticAngle <= 180 - majorDefocus > minorDefocus - 0 <= phaseShift <= 2*pi - maxResolution > 0 - rawAngle matches mdoc angle as defined above - 0 <= zIndex <= (z-Dimension of tilt series - 1)

Bento007 changed the title ~~tsmith/1528-frame-ctf-valdiation~~ chore: Update output data validations for frames and ctf Apr 16, 2025

Bento007 added 2 commits April 18, 2025 12:39

Merge branch 'refs/heads/main' into tsmith/1528-frame-ctf-valdiation

9dff442

move tests to get correct metadata

568ebdf

Bento007 added 2 commits April 18, 2025 13:20

is in_plane_rotation for test_mdoc_tilt_axis_angle_in_alignment_per_s…

dc68a97

…ection_alignment_parameters

make suggestest changes

9cfe355

Bento007 added 8 commits April 18, 2025 16:52

use angle_helper

56e70ee

use 1 tolerance for dose rate

bc467e1

use 1 tolerance for dose rate

976a1bc

use 1 tolerance for dose rate

8771009

minor tweak

407329c

Raise AssertionError if multiple errors are found

01d511d

use copies when distructive operations happen on the data

d661774

undo .copy

2b35638

Bento007 requested review from manasaV3, Copilot and uermel and removed request for manasaV3 April 21, 2025 20:59

Copilot AI reviewed Apr 21, 2025

View reviewed changes

Apply suggestions from code review

dadfb58

Co-authored-by: Copilot <[email protected]>

Bento007 closed this Apr 21, 2025

Bento007 reopened this Apr 21, 2025

manasaV3 reviewed Apr 22, 2025

View reviewed changes

Bento007 requested a review from manasaV3 April 24, 2025 23:47

Merge branch 'refs/heads/main' into tsmith/1528-frame-ctf-valdiation

84b1885

Signed-off-by: Bento007 <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Update output data validations for frames and ctf #484

chore: Update output data validations for frames and ctf #484

Bento007 commented Apr 16, 2025 •

edited

Loading

Bento007 commented Apr 17, 2025

Bento007 commented Apr 17, 2025 •

edited

Loading

uermel commented Apr 18, 2025

uermel commented Apr 18, 2025

Bento007 commented Apr 18, 2025

uermel commented Apr 18, 2025

10014

Copilot AI left a comment

Bento007 commented Apr 21, 2025

manasaV3 Apr 22, 2025

manasaV3 Apr 22, 2025

Bento007 Apr 24, 2025

manasaV3 Apr 22, 2025

Bento007 Apr 24, 2025 •

edited

Loading

manasaV3 Apr 22, 2025

manasaV3 Apr 22, 2025

	if frame_metadata.get("is_gain_corrected") is False: # todo is none considered false?
	if not frame_metadata.get("is_gain_corrected"):

chore: Update output data validations for frames and ctf #484

Are you sure you want to change the base?

chore: Update output data validations for frames and ctf #484

Conversation

Bento007 commented Apr 16, 2025 • edited Loading

Description

Validations for MDOC/Frames

Validations for Frames Metadata

Validations for PerSectionParameter Metadata

Extra

Bento007 commented Apr 17, 2025

Bento007 commented Apr 17, 2025 • edited Loading

uermel commented Apr 18, 2025

uermel commented Apr 18, 2025

Bento007 commented Apr 18, 2025

10014

uermel commented Apr 18, 2025

10014

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Bento007 commented Apr 21, 2025

manasaV3 Apr 22, 2025

Choose a reason for hiding this comment

manasaV3 Apr 22, 2025

Choose a reason for hiding this comment

Bento007 Apr 24, 2025

Choose a reason for hiding this comment

manasaV3 Apr 22, 2025

Choose a reason for hiding this comment

Bento007 Apr 24, 2025 • edited Loading

Choose a reason for hiding this comment

manasaV3 Apr 22, 2025

Choose a reason for hiding this comment

manasaV3 Apr 22, 2025

Choose a reason for hiding this comment

Bento007 commented Apr 16, 2025 •

edited

Loading

Bento007 commented Apr 17, 2025 •

edited

Loading

Bento007 Apr 24, 2025 •

edited

Loading