Skip to content

Commit 940be4f

Browse files
committed
New version 2 of V&V docs.
Use the python script to prepare for a new filter that is being worked on. Tested on ComputeGroupingDensityFilter. Changes will probably be needed.
1 parent 657f6c2 commit 940be4f

15 files changed

Lines changed: 605 additions & 3132 deletions
Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# Deviations from DREAM3D 6.5.171: <FilterName>
2+
3+
This file lists every documented behavioral difference between this SIMPLNX filter and its DREAM3D 6.5.171 equivalent.
4+
5+
Entries are referenced by stable ID (`<FilterName>-D<N>`) from the V&V report and from public migration guidance. The ID is stable across renames; the Filter UUID field is the permanent cross-reference anchor.
6+
7+
---
8+
9+
## <FilterName>-D1
10+
11+
| Field | Value |
12+
|---|---|
13+
| **Deviation ID** | `<FilterName>-D1` |
14+
| **Filter UUID** | `<uuid>` |
15+
| **Status** | active *or* superseded by `<FilterName>-D<M>` *or* retired YYYY-MM-DD |
16+
17+
**Symptom:** *<one-sentence user-visible symptom>*
18+
19+
**Root cause:** *bug | precision | order of operations | library | algorithmic choice*
20+
*One paragraph explaining the technical mechanism. Cite source files and line ranges where helpful.*
21+
22+
**Affected users:** *<who actually sees this — e.g., "anyone with features spanning image boundaries", "only Hex-symmetry datasets", "datasets larger than 10M voxels">*
23+
24+
**Recommendation:** *trust SIMPLNX | trust 6.5.171 | either acceptable within tolerance X | see quick-patch link for legacy-parity*
25+
*One sentence justifying the recommendation.*
26+
27+
---
28+
29+
## <FilterName>-D2
30+
31+
| Field | Value |
32+
|---|---|
33+
| **Deviation ID** | `<FilterName>-D2` |
34+
| **Filter UUID** | `<uuid>` |
35+
| **Status** | active |
36+
37+
**Symptom:** ...
38+
39+
**Root cause:** ...
40+
41+
**Affected users:** ...
42+
43+
**Recommendation:** ...
44+
45+
---
46+
47+
## Examples (delete this section before sign-off)
48+
49+
### Precision example
50+
51+
| Field | Value |
52+
|---|---|
53+
| **Deviation ID** | `ComputeEulerAngles-D1` |
54+
| **Filter UUID** | `aaaa1111-0000-0000-0000-000000000001` *(illustrative)* |
55+
| **Status** | active |
56+
57+
**Symptom:** Euler-angle output differs from 6.5.171 by up to 0.003° in orientations near grain boundaries.
58+
59+
**Root cause:** Precision. SIMPLNX performs the internal orientation-matrix operations in `double`; 6.5.171 performed the same operations in `float`.
60+
61+
**Affected users:** Workflows that compute orientation statistics on features larger than ~10⁴ voxels, where accumulated float32 round-off becomes visible at the 10⁻³ degree level. Users who only visualize IPF colors will not notice.
62+
63+
**Recommendation:** Trust SIMPLNX. The 6.5.171 output was limited by float32 round-off and is not materially more correct for any downstream calculation.
64+
65+
### Legacy-bug example
66+
67+
| Field | Value |
68+
|---|---|
69+
| **Deviation ID** | `SegmentFeatures-D2` |
70+
| **Filter UUID** | `aaaa2222-0000-0000-0000-000000000002` *(illustrative)* |
71+
| **Status** | active |
72+
73+
**Symptom:** FeatureId count on a 50×50×50 block test pattern is 27 in SIMPLNX and 26 in 6.5.171.
74+
75+
**Root cause:** Bug in 6.5.171. The outer segmentation loop used `< dimZ` where it should have used `<= dimZ`, silently dropping features that touched the +Z boundary. Corrected in SIMPLNX.
76+
77+
**Affected users:** Anyone who ran `SegmentFeatures` on datasets where a feature touched the +Z volume boundary. The missing feature was always the one nearest +Z.
78+
79+
**Recommendation:** Trust SIMPLNX. The 6.5.171 result was mathematically incorrect.

docs/vv_templates/mtr_filter_verification_validation.md

Lines changed: 0 additions & 341 deletions
This file was deleted.

docs/vv_templates/oracle_classes_quick_reference.md

Lines changed: 0 additions & 48 deletions
This file was deleted.

docs/vv_templates/oracle_discussion.md

Lines changed: 0 additions & 76 deletions
This file was deleted.
Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Exemplar Archive Provenance: <archive_name>.tar.gz
2+
3+
This sidecar records how an exemplar archive used in unit tests was generated. It is the answer to "where did this gold-standard data come from?"
4+
5+
One sidecar per archive. The archive name and SHA512 must match `download_test_data()` in `src/Plugins/<P>/test/CMakeLists.txt`.
6+
7+
---
8+
9+
## Archive identity
10+
11+
| Field | Value |
12+
|---|---|
13+
| **Archive** | `<archive_name>.tar.gz` |
14+
| **SHA512** | `<copy from test/CMakeLists.txt>` |
15+
| **Used by tests** | `<TestCaseName1>`, `<TestCaseName2>` |
16+
| **Generated by** | *<engineer name>* |
17+
| **Generated on** | *YYYY-MM-DD* |
18+
| **Generated at commit** | *<commit hash where the generating pipeline / script lived>* |
19+
20+
## How it was generated
21+
22+
*Describe in 2–4 sentences how the archive's contents were produced. Reference any pipeline (`.d3dpipeline`) or script (`.py`, `.m`) by file path. State whether inputs were from a public dataset, synthetic generation, or hand-construction.*
23+
24+
Example:
25+
> The archive contains a 100×100×100 synthetic microstructure generated by `pipelines/generate_in100_subvol.d3dpipeline` plus the expected output of `ComputeGroupingDensityFilter` on that input. Inputs were generated synthetically (no external data); expected output was computed by the Class 3 Rowenhorst 2015 §4.2 worked example.
26+
27+
## Canonical oracle output
28+
29+
*Which arrays in the archive are the canonical oracle output (the thing tests compare against)? Reference by DataPath.*
30+
31+
| DataPath | Source of expected values |
32+
|---|---|
33+
| `/<Group>/<Array>` | *Class 1 hand derivation* / *Class 2 script X.py* / *Class 3 paper ref* / *Class 4 invariant* / *Class 5 expert sign-off* |
34+
35+
## Oracle provenance (Classes 2, 3, 5 only)
36+
37+
*Classes 1 and 4 need no provenance block — the oracle lives in the test code directly.*
38+
39+
### Class 2 — Reference implementation
40+
41+
- **Library:** *<name>*
42+
- **Exact version:** *<x.y.z (release date)>*
43+
- **Runtime:** *<e.g., Python 3.11, Octave 8.2.0>*
44+
- **Random seed:** *<seed, if any>*
45+
- **Script in archive:** *<filename.py / filename.m>*
46+
- *Optional:* **Output hash for drift detection:** *<sha256>*
47+
48+
### Class 3 — Paper-based
49+
50+
- **Citation:** *<authors, title, journal, vol/issue, pages, year>*
51+
- **DOI:** *<doi>*
52+
- **Edition:** *<edition #>*
53+
- **Figure / Table / Equation #:** *<reference>*
54+
- **Page #:** *<page>*
55+
- **Paper PDF in archive:** *`notes/<short_name>.pdf`*
56+
- **Reproduced figure (if any):** *`notes/<short_name>_figN.png`*
57+
58+
### Class 5 — Expert-visual
59+
60+
- **Approving expert:** *<name>*
61+
- **Approval date:** *YYYY-MM-DD*
62+
- **Signed-off outputs:** *`notes/expert_signoff/*.png`*
63+
- **Class-5-only justification:** *Why no Class 1–4 oracle was feasible for this filter.*
64+
65+
## Second-engineer oracle review
66+
67+
- **Reviewer:** *<name>* OR *skipped*
68+
- **Date:** *YYYY-MM-DD*
69+
- **Skip reason** (if skipped): *<reason — typically "no second engineer realistically available">*
70+
71+
## Regenerated to fix a circular-oracle situation?
72+
73+
*If this archive was created to *replace* a prior archive that was regenerated from post-fix SIMPLNX output (a circular oracle), state that here with a reference to the prior archive.*
74+
75+
> *Example: this archive replaces `compute_grouping_density.tar.gz` (retired YYYY-MM-DD), which was regenerated from SIMPLNX output in `<PR#>` after the bug fix in `<PR#>` and therefore could not be used as an independent oracle.*

docs/vv_templates/report_gates.md

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# V&V Report — Section Gates
2+
3+
A section of `src/Plugins/<P>/vv/<FilterName>.md` is "done" when all its gates pass. Sections may be worked in any order.
4+
5+
**The only ordering rule:** the **Oracle** is chosen *before* any DREAM3D 6.5.171 comparison is run. 6.5.171 is never the source of truth — it is the *thing being compared against* the independently-established oracle.
6+
7+
---
8+
9+
## Header table
10+
11+
- [ ] Plugin, SIMPLNX UUID, legacy DREAM3D equivalent (or "None") filled in
12+
- [ ] Status reflects current state: `DRAFT` | `READY FOR REVIEW` | `COMPLETE`
13+
- [ ] Verified-commit field present (filled in at SBIR deliverable assembly)
14+
- [ ] Sign-off line has named engineer(s) and date at sign-off
15+
16+
## Summary
17+
18+
- [ ] 2–3 sentences only
19+
- [ ] States what the filter does
20+
- [ ] States the verification approach (one phrase, e.g., "Class 3 paper-based vs Rowenhorst 2015")
21+
- [ ] States the headline result (e.g., "1 deviation, all tests pass")
22+
23+
## Algorithm Relationship
24+
25+
- [ ] One classification: Port | Minor changes | Rewrite | New filter
26+
- [ ] One-line evidence (UUID inheritance, PR history, line-count diff with legacy)
27+
- [ ] **Rewrite + outputs diverge from legacy** → explicit defense required in the Deviations file (same UUID is a claim of functional equivalence)
28+
29+
## Oracle
30+
31+
- [ ] Class named (1–5)
32+
- 1 = Analytical (closed-form expected output on toy input)
33+
- 2 = Reference implementation (NumPy / SciPy / MTEX / EbsdLib upstream)
34+
- 3 = Paper-based (published figure / table / equation)
35+
- 4 = Invariant-based (derivable property the output must satisfy)
36+
- 5 = Expert-visual (last resort, requires justification)
37+
- [ ] If Class 5: justification block stating why no Class 1–4 oracle was feasible
38+
- [ ] One-line description of how oracle was applied
39+
- [ ] Encoded test reference: `<file>::<TEST_CASE>` exists and is greppable
40+
- [ ] N fixtures stated; all pass at the verified commit
41+
- [ ] Second-engineer review of oracle design, OR documented skip reason
42+
43+
## Code path coverage
44+
45+
- [ ] All algorithm code paths enumerated (kernel choice, mask on/off, edge cases, error paths)
46+
- [ ] Each path maps to ≥1 test case
47+
- [ ] Parameter-dependent paths: every combination of interest represented
48+
49+
## Test inventory
50+
51+
- [ ] Every `TEST_CASE` in the filter's test file listed
52+
- [ ] Each marked: `kept` | `new-for-V&V` | `retired` (with one-line reason for retired)
53+
- [ ] All non-retired tests pass at the verified commit in **both** in-core and OOC builds
54+
55+
## Exemplar archive
56+
57+
- [ ] Archive name matches `download_test_data()` entry in `test/CMakeLists.txt`
58+
- [ ] SHA512 in report matches SHA512 in `test/CMakeLists.txt`
59+
- [ ] Provenance sidecar exists at `src/Plugins/<P>/vv/provenance/<archive>.md` and documents:
60+
- who generated the archive
61+
- when
62+
- with what pipeline / script
63+
- what oracle output was canonical
64+
- [ ] If the archive was regenerated during V&V to fix a circular-oracle situation → documented in the sidecar
65+
66+
## Deviations from DREAM3D 6.5.171
67+
68+
- [ ] Comparison was run on at least one fixture (named in the report)
69+
- [ ] If no deviations: a one-line confirmation that 6.5.171 and SIMPLNX outputs matched within tolerance
70+
- [ ] If deviations: each ID referenced in the report points to a fleshed-out entry in `src/Plugins/<P>/vv/deviations/<FilterName>.md`
71+
- [ ] Each deviation entry has:
72+
- stable ID (`<FilterName>-D<N>`)
73+
- filter UUID (permanent cross-reference anchor)
74+
- symptom (user-visible)
75+
- root cause: `bug` | `precision` | `order of operations` | `library` | `algorithmic choice`
76+
- affected users
77+
- recommendation: `trust SIMPLNX` | `trust 6.5.171` | `either acceptable within tolerance` | `see quick-patch link`

0 commit comments

Comments
 (0)