Skip to content

FF_Benchmark.md

Latest commit

 

History

History
215 lines (166 loc) · 8.49 KB

FF_Benchmark.md

File metadata and controls

215 lines (166 loc) · 8.49 KB

FF-HEDM Benchmark Test

MIDAS v10+

Overview

The FF-HEDM benchmark (tests/test_ff_hedm.py) is an automated end-to-end test that validates the full FF-HEDM pipeline by:

  1. Running a forward simulation to generate synthetic diffraction data from known grain orientations
  2. Enriching the output Zarr with analysis metadata
  3. Running the complete ff_MIDAS.py pipeline (peaksearch → indexing → refinement → grain list)
  4. Verifying that the pipeline completes successfully and recovers the input grains

Pipeline Data Flow

flowchart TD
    subgraph Inputs["📂 Input Files"]
        PARAMS["Parameters.txt"]
        GRAINS_IN["GrainsSim.csv<br/>(3 Au grains)"]
    end

    subgraph Step1["Step 1: Forward Simulation"]
        PARSE["Parse & rewrite paths"]
        FWD["ForwardSimulationCompressed"]
        PARAMS --> PARSE
        GRAINS_IN --> PARSE
        PARSE -->|"test_Parameters.txt"| FWD
    end

    subgraph Step1Out["Simulation Outputs"]
        ZIP_RAW["Au_FF_*_scanNr_0.zip<br/>(Zarr: 1440 frames)"]
        SPOT_MAT["SpotMatrixGen.csv<br/>(theoretical spot positions)"]
        GRAINS_GEN["GrainsGen.csv<br/>(re-indexed grain list)"]
        HKLS_SIM["hkls.csv<br/>(HKL reflections)"]
    end

    FWD --> ZIP_RAW
    FWD --> SPOT_MAT
    FWD --> GRAINS_GEN
    FWD --> HKLS_SIM

    subgraph Step2["Step 2: Zarr Enrichment"]
        ENRICH["write_analysis_parameters<br/>(inject metadata)"]
        RENAME["Rename to<br/>*.analysis.MIDAS.zip"]
    end

    ZIP_RAW --> ENRICH
    PARAMS --> ENRICH
    ENRICH --> RENAME

    subgraph Step3["Step 3: ff_MIDAS.py Pipeline"]
        direction TB
        HKLGEN["GetHKLListZarr"]
        PEAKS["PeaksFittingOMPZarrRefactor<br/>→ find spots in images"]
        MERGE["MergeOverlappingPeaksAllZarr<br/>→ consolidate peaks"]
        CALCR["CalcRadiusAllZarr<br/>→ compute ring radii"]
        FITS["FitSetupZarr<br/>→ lab coordinate transform"]
        BIND["SaveBinData<br/>→ binary format"]
        INDEX["IndexerOMP<br/>→ match spots to orientations"]
        REFINE["FitPosOrStrainsOMP<br/>→ refine positions & strains"]
        PROCG["ProcessGrainsZarr<br/>→ final grain list"]

        HKLGEN --> PEAKS
        PEAKS --> MERGE
        MERGE --> CALCR
        CALCR --> FITS
        FITS --> BIND
        BIND --> INDEX
        INDEX --> REFINE
        REFINE --> PROCG
    end

    RENAME -->|"*.analysis.MIDAS.zip"| HKLGEN

    subgraph Step3Out["Pipeline Outputs in LayerNr_1/"]
        INPUTALL["InputAll.csv<br/>(transformed spot data)"]
        SPOTM["SpotMatrix.csv<br/>(matched spots)"]
        RESULT["Result_*.csv<br/>(indexing results)"]
        GRAINS_OUT["Grains.csv<br/>(%NumGrains 3)"]
    end

    PEAKS -->|"Temp/<br/>(per-frame peaks)"| MERGE
    FITS --> INPUTALL
    INDEX --> RESULT
    PROCG --> GRAINS_OUT
    REFINE -->|"Output/<br/>(refined orientations)"| PROCG

    subgraph Step4["Step 4: Validation"]
        CHECK["✅ Exit code 0<br/>NumGrains == 3"]
    end

    GRAINS_OUT --> CHECK

    style Inputs fill:#e8f4f8,stroke:#2196F3
    style Step1 fill:#fff3e0,stroke:#FF9800
    style Step1Out fill:#fce4ec,stroke:#E91E63
    style Step2 fill:#f3e5f5,stroke:#9C27B0
    style Step3 fill:#e8f5e9,stroke:#4CAF50
    style Step3Out fill:#fce4ec,stroke:#E91E63
    style Step4 fill:#e8f5e9,stroke:#4CAF50
    style CHECK fill:#c8e6c9,stroke:#2E7D32
Loading

Prerequisites

  • MIDAS must be compiled (all FF_HEDM binaries built)
  • The midas_env conda environment must be active: 
    source /path/to/miniconda3/bin/activate midas_env

Usage

python tests/test_ff_hedm.py [-nCPUs N] [-paramFN /path/to/Parameters.txt] [--px-overlap] [--dual-dataset] [--no-cleanup] [--cleanup-only]

Arguments

Argument Default Description
-nCPUs 1 Number of CPUs for parallel steps
-paramFN FF_HEDM/Example/Parameters.txt Path to the parameter file
--px-overlap off Also run pixel-overlap peaksearch test using Parameters_px_overlaps.txt
--dual-dataset off Also run dual-dataset refinement test using ff_dual_datasets.py
--no-cleanup off Keep all generated files after the test
--cleanup-only off Only cleanup generated files, don't run any tests

Example

source ~/miniconda3/bin/activate midas_env
python ~/opt/MIDAS/tests/test_ff_hedm.py -nCPUs 4

# With pixel-overlap test:
python ~/opt/MIDAS/tests/test_ff_hedm.py -nCPUs 4 --px-overlap

# Keep files for inspection:
python ~/opt/MIDAS/tests/test_ff_hedm.py -nCPUs 4 --no-cleanup

# Just cleanup leftover files:
python ~/opt/MIDAS/tests/test_ff_hedm.py --cleanup-only

Working Directory

The benchmark always runs in FF_HEDM/Example/ relative to MIDAS_HOME. All intermediate and output files are generated there.

What It Does

Step 1: Forward Simulation

Runs ForwardSimulationCompressed with the example parameter file and GrainsSim.csv (3 Au grains) to produce a synthetic Zarr zip containing detector images.

Step 2: Zarr Enrichment

Injects analysis metadata (ring thresholds, omega ranges, box sizes, etc.) into the Zarr zip so that ff_MIDAS.py can process it directly.

Step 3: Pipeline Execution

Launches ff_MIDAS.py with -convertFiles 0 (skipping raw data conversion since the Zarr is already prepared). The pipeline runs:

  • PeakSearch → finds diffraction spots in the synthetic images
  • MergeOverlappingPeaks → consolidates overlapping peaks
  • CalcRadius → computes ring radii
  • FitSetup → transforms data for indexing
  • IndexerOMP → indexes spots to grain orientations
  • FitPosOrStrainsOMP → refines grain positions and strains
  • ProcessGrains → produces the final Grains.csv

Step 4: Regression Comparison

Compares the newly generated *_consolidated.h5 against the reference file consolidated_Output.h5, checking each pipeline stage (PeaksFitting, MergeOverlaps, CalcRadius, FitSetup, Grains).

Step 5 (Optional): Pixel-Overlap Test

When --px-overlap is specified, the test additionally:

  1. Patches UsePixelOverlap 1 and doPeakFit 1 into the existing Zarr zip
  2. Re-runs ff_MIDAS.py with the pixel-overlap parameters
  3. Compares the output against consolidated_Output_px_overlaps.h5

This validates the pixel-overlap merge path end-to-end.

Step 6 (Optional): Dual-Dataset Test

When --dual-dataset is specified, the test additionally:

  1. Runs ff_dual_datasets.py using the same simulated data as both datasets (with zero offsets)
  2. Verifies that MapDatasets and FitPosOrStrainsDoubleDataset complete successfully
  3. Compares the dual-dataset Grains.csv against the single-dataset results

This validates the dual-dataset pipeline end-to-end, including the dataset mapping and combined refinement steps.

Expected Output

A successful run produces output like:

First line of Grains.csv: %NumGrains 3
*** Automated FF_HEDM Benchmark Suite Executed Successfully ***

The benchmark should recover all 3 input grains from GrainsSim.csv.

Generated Files

The following files are generated in FF_HEDM/Example/ and are excluded via .gitignore:

File/Directory Description
test_Parameters.txt Modified parameter file with absolute paths
Au_FF_*.analysis.MIDAS.zip Zarr zip with synthetic diffraction data
GrainsGen.csv Re-indexed grain list from simulation
SpotMatrixGen.csv Theoretical spot positions from simulation
hkls.csv HKL list generated during simulation
LayerNr_1/ Full pipeline output directory
runinfo/ Parsl runtime information

Troubleshooting

  • ForwardSimulationCompressed not found: Run cmake --build . --target ForwardSimulationCompressed in the build/ directory.
  • zarr.ZipStore error: Ensure you are using the midas_env conda environment which has the correct zarr version.
  • Pipeline fails at indexing: Check that the simulation produced valid detector images. The SpotMatrixGen.csv file can be inspected to verify spot positions.