feat(deploy): configurable threshold with metadata.json#3596
Draft
ashwinvaidya17 wants to merge 3 commits into
Draft
feat(deploy): configurable threshold with metadata.json#3596ashwinvaidya17 wants to merge 3 commits into
ashwinvaidya17 wants to merge 3 commits into
Conversation
Signed-off-by: Ashwin Vaidya <ashwin.vaidya@intel.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces deploy-time configurable thresholding by adding image/pixel sensitivity overrides that can be provided at inference time, and by exporting a metadata.json sidecar containing default post-processing parameters (incl. sensitivities). This aligns with the deploy/export pipeline by making exported artifacts more self-describing and enabling inferencers to pick up defaults automatically.
Changes:
- Extend
AnomalibModule/PostProcessorforward paths to accept optionalimage_sensitivityandpixel_sensitivityoverrides. - Write a
metadata.jsonsidecar during Torch/ONNX/OpenVINO export and add a small metadata load/dump + migration utility. - Update Torch/OpenVINO inferencers to read
metadata.jsonand allow constructor/per-call sensitivity overrides; add unit tests for metadata and the new post-processor threshold logic.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/post_processing/test_post_processor.py | Adds unit coverage for effective-threshold computation and sensitivity overrides in PostProcessor.forward. |
| tests/unit/deploy/test_metadata.py | Adds unit coverage for metadata load/dump, validation errors, and migration behavior. |
| src/anomalib/post_processing/post_processor.py | Adds sensitivity override inputs, effective-threshold computation, and alters thresholding behavior in forward. |
| src/anomalib/post_processing/mebin_post_processor.py | Adjusts forward signature to accept sensitivity args for interface compatibility. |
| src/anomalib/models/components/base/export_mixin.py | Adds metadata.json writing during export and extends ONNX export to include sensitivity inputs. |
| src/anomalib/models/components/base/anomalib_module.py | Extends model forward to accept sensitivity overrides and forwards them to the post-processor. |
| src/anomalib/deploy/metadata/migration.py | Introduces schema versioning and a legacy-to-v1 migration path. |
| src/anomalib/deploy/metadata/_core.py | Implements metadata load/dump/validation helpers and a SchemaValidationError. |
| src/anomalib/deploy/metadata/init.py | Exposes the metadata public API (load_metadata, dump_metadata, etc.). |
| src/anomalib/deploy/inferencers/torch_inferencer.py | Adds metadata-driven default sensitivities and per-call overrides for Torch inference. |
| src/anomalib/deploy/inferencers/openvino_inferencer.py | Adds metadata-driven default sensitivities and per-call overrides for OpenVINO inference. |
Signed-off-by: Ashwin Vaidya <ashwin.vaidya@intel.com>
Signed-off-by: Ashwin Vaidya <ashwin.vaidya@intel.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.
Comments suppressed due to low confidence (2)
src/anomalib/post_processing/post_processor.py:264
- When normalization stats are unavailable (
image_min/image_maxorpixel_min/pixel_maxare NaN),_normalize()returns the raw predictions unchanged. In that case,forward()still thresholds using a fixed 0.5 (the normalized-space boundary), which will silently apply the wrong decision boundary to unnormalized scores/maps. Consider gating the 0.5 thresholding on normalization actually being applied (stats present), otherwise fall back to raw thresholds (or skip thresholding).
if self.enable_thresholding:
if self.enable_normalization:
threshold = torch.tensor(0.5, device=self.image_threshold.device, dtype=self.image_threshold.dtype)
pred_label = self._apply_threshold(pred_score, threshold)
pred_mask = self._apply_threshold(anomaly_map, threshold)
else:
pred_label = self._apply_threshold(pred_score, self.image_threshold)
pred_mask = self._apply_threshold(anomaly_map, self.pixel_threshold)
src/anomalib/post_processing/post_processor.py:358
_apply_thresholdno longer guards againstthresholdbeing NaN. Since thresholds are initialized as NaN buffers and may remain NaN if calibration/threshold fitting was not run,preds > thresholdwill produce all-False labels/masks, which is a silent failure mode. Consider explicitly handling NaN thresholds (e.g., returnNoneto indicate "no threshold available" or raise a clear error).
Returns:
torch.Tensor | None: Thresholded predictions or None if input is None.
"""
if preds is None:
return preds
return preds > threshold
Comment on lines
238
to
+252
| if self.enable_normalization: | ||
| pred_score = self._normalize(pred_score, self.image_min, self.image_max, self.image_threshold) | ||
| anomaly_map = self._normalize(predictions.anomaly_map, self.pixel_min, self.pixel_max, self.pixel_threshold) | ||
| image_eff_threshold = self._effective_threshold( | ||
| self.image_threshold, | ||
| self.image_min, | ||
| self.image_max, | ||
| image_sens, | ||
| ) | ||
| pixel_eff_threshold = self._effective_threshold( | ||
| self.pixel_threshold, | ||
| self.pixel_min, | ||
| self.pixel_max, | ||
| pixel_sens, | ||
| ) | ||
| pred_score = self._normalize(pred_score, self.image_min, self.image_max, image_eff_threshold) | ||
| anomaly_map = self._normalize(predictions.anomaly_map, self.pixel_min, self.pixel_max, pixel_eff_threshold) |
Comment on lines
+308
to
+319
| try: | ||
| predictions = self.model( | ||
| image, | ||
| torch.tensor(img_sens, device=self.device), | ||
| torch.tensor(pix_sens, device=self.device), | ||
| ) | ||
| except TypeError: | ||
| logger.warning( | ||
| "Image only API is deprecated and will be removed in Anomalib 2.7.0." | ||
| " Models exported from current version of Anomalib already support the new API.", | ||
| ) | ||
| predictions = self.model(image) |
Comment on lines
+133
to
+147
| self.metadata = self._load_metadata(path) | ||
| self._default_image_sensitivity = ( | ||
| image_sensitivity | ||
| if image_sensitivity is not None | ||
| else self.metadata.get("postprocess", {}).get("image_sensitivity", 0.5) | ||
| if self.metadata | ||
| else 0.5 | ||
| ) | ||
| self._default_pixel_sensitivity = ( | ||
| pixel_sensitivity | ||
| if pixel_sensitivity is not None | ||
| else self.metadata.get("postprocess", {}).get("pixel_sensitivity", 0.5) | ||
| if self.metadata | ||
| else 0.5 | ||
| ) |
Comment on lines
+174
to
+179
| def forward( | ||
| self, | ||
| batch: torch.Tensor, | ||
| image_sensitivity: torch.Tensor | None = None, | ||
| pixel_sensitivity: torch.Tensor | None = None, | ||
| ) -> InferenceBatch: |
| if _parse_version(metadata["schema_version"]) > _parse_version(CURRENT_SCHEMA_VERSION): | ||
| logger.warning( | ||
| "Metadata schema %s is newer than supported (%s). " | ||
| "Unknown fields will be ignored. Consider upgrading anomalib.", |
Comment on lines
79
to
+91
| def to_torch( | ||
| self, | ||
| export_root: Path | str, | ||
| model_file_name: str = "model", | ||
| write_metadata: bool = True, | ||
| ) -> Path: | ||
| """Export model to PyTorch format. | ||
|
|
||
| Args: | ||
| export_root (Path | str): Path to the output folder | ||
| model_file_name (str): Name of the exported model | ||
| write_metadata (bool): Whether to write metadata.json sidecar file. | ||
| Defaults to ``True``. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Description
✨ Changes
Select what type of change your PR is:
✅ Checklist
Before you submit your pull request, please make sure you have completed the following steps:
For more information about code review checklists, see the Code Review Checklist.