fix(analyzer): keep recognizer score thresholds when config omits them by omri374 · Pull Request #2210 · data-privacy-stack/presidio

omri374 · 2026-08-04T08:42:24Z

Change Description

Recognizer score thresholds set by a recognizer class are discarded when the registry configuration omits score_thresholds.

RecognizerListLoader.get() excludes score_thresholds from the constructor kwargs and instead assigns it after instantiation:

score_thresholds = normalize_score_thresholds(recognizer_conf.get("score_thresholds"))
...
recognizer = recognizer_cls(**kwargs)
recognizer.score_thresholds = score_thresholds

normalize_score_thresholds(None) returns {}, so an entry that says nothing about thresholds overwrites whatever the class set for itself.

This is the opposite of how every other configuration key behaves. Keys such as context are only passed when present, so context = context if context else self.CONTEXT keeps the class default. Omission means "say nothing", not "reset to empty".

Fix

Apply the configured value only when the key is present. An explicit empty mapping remains a deliberate reset.

if has_score_thresholds:
    recognizer.score_thresholds = score_thresholds

Applied in three places: the predefined branch and the custom branch of RecognizerListLoader.get(), and RecognizerRegistry.add_pattern_recognizer_from_dict.

Impact

No predefined recognizer currently declares its own thresholds, so the assignment has always been {} over {} and the bug has never been observable. It becomes observable the moment a recognizer class defines a threshold as part of its own definition, which #2159 is the first to do: all six of its new recognizers load with score_thresholds={} and emit pattern-only matches that their tests assert are suppressed.

Tests

Four regression tests using a recognizer that declares its own thresholds:

omitted key keeps the class default
explicit value overrides the class default
explicit empty mapping clears the thresholds
the dict path follows the same rule

The first and last fail without the fix.

Issue reference

Follow-up to #2116. Blocks #2159.

Checklist

I have reviewed the contribution guidelines
I agree to follow this project's Code of Conduct
I confirm that I have the right to submit this contribution and that it does not knowingly contain proprietary or confidential code.
My code includes unit tests
All unit tests and lint checks pass locally
My PR contains documentation updates / additions if required

RecognizerListLoader assigned the normalized score_thresholds value unconditionally after instantiating a recognizer. normalize_score_thresholds maps a missing key to an empty mapping, so a configuration entry that said nothing about thresholds silently overwrote whatever the recognizer class had set for itself. Every other configuration key behaves the opposite way: absent keys are never passed to the constructor, so class defaults stand. Thresholds are now applied only when the key is present. An explicit empty mapping still clears them. No predefined recognizer currently declares its own thresholds, so this is a no-op for the existing suite, but it blocks any recognizer that wants a threshold as part of its own definition. The same unconditional assignment in the custom-recognizer branch and in add_pattern_recognizer_from_dict is fixed for consistency.

github-actions · 2026-08-04T08:43:48Z

Coverage report (presidio-structured)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
presidio-structured/presidio_structured/data
data_processors.py
Project Total

_{This report was generated by python-coverage-comment-action}

github-actions · 2026-08-04T08:43:51Z

Coverage report (presidio-cli)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
presidio-cli/presidio_cli
cli.py
Project Total

_{This report was generated by python-coverage-comment-action}

Fixed various issues with recognizer configurations and behavior, including score thresholds, region codes, and language model parameters.

github-actions · 2026-08-04T08:44:55Z

Coverage report (presidio-anonymizer)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
presidio-anonymizer/presidio_anonymizer
__init__.py
anonymizer_engine.py
presidio-anonymizer/presidio_anonymizer/entities/engine
pii_entity.py
presidio-anonymizer/presidio_anonymizer/entities/engine/result
operator_result.py
presidio-anonymizer/presidio_anonymizer/operators
custom.py
Project Total

_{This report was generated by python-coverage-comment-action}

Copilot

Pull request overview

Fixes a configuration-loading bug in presidio-analyzer where a recognizer’s class-defined score_thresholds were unintentionally overwritten with {} when the registry config omitted the score_thresholds key (because normalize_score_thresholds(None) returns {} and the loader assigned it unconditionally).

Changes:

Update RecognizerListLoader.get() to only assign recognizer.score_thresholds when the config key is present (predefined + custom branches).
Update RecognizerRegistry.add_pattern_recognizer_from_dict() to apply the same “only if key present” rule.
Add regression tests covering omission vs explicit override vs explicit empty mapping behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
presidio-analyzer/presidio_analyzer/recognizer_registry/recognizers_loader_utils.py	Preserve class-defined thresholds unless configuration explicitly provides `score_thresholds`.
presidio-analyzer/presidio_analyzer/recognizer_registry/recognizer_registry.py	Apply the same omission semantics for the dict-based recognizer loading path.
presidio-analyzer/tests/test_recognizer_registry.py	Add regression tests ensuring omission keeps class defaults while explicit values override/clear.
CHANGELOG.md	Adds a changelog entry for this fix (but see PR comment re: repo policy).

github-actions · 2026-08-04T08:46:41Z

Coverage report (presidio-image-redactor)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
presidio-image-redactor/presidio_image_redactor
dicom_image_pii_verify_engine.py
document_intelligence_ocr.py
image_analyzer_engine.py
Project Total

_{This report was generated by python-coverage-comment-action}

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

github-actions · 2026-08-04T08:49:07Z

Coverage report (presidio-analyzer)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
presidio-analyzer/presidio_analyzer
analyzer_engine.py
entity_recognizer.py
presidio-analyzer/presidio_analyzer/chunkers
text_chunker_provider.py
presidio-analyzer/presidio_analyzer/context_aware_enhancers
lemma_context_aware_enhancer.py
presidio-analyzer/presidio_analyzer/input_validation
schemas.py
yaml_recognizer_models.py
presidio-analyzer/presidio_analyzer/llm_utils
config_loader.py
presidio-analyzer/presidio_analyzer/nlp_engine
__init__.py
nlp_engine_provider.py
presidio-analyzer/presidio_analyzer/predefined_recognizers
__init__.py
presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific
__init__.py
presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/finland
fi_personal_identity_code_recognizer.py
presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/germany
de_bsnr_recognizer.py
de_id_card_recognizer.py
de_lanr_recognizer.py
de_passport_recognizer.py
de_social_security_recognizer.py
de_vat_id_recognizer.py
presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/poland
pl_pesel_recognizer.py
presidio-analyzer/presidio_analyzer/predefined_recognizers/ner
gliner_recognizer.py
huggingface_ner_recognizer.py
presidio-analyzer/presidio_analyzer/predefined_recognizers/third_party
azure_ai_language.py
presidio-analyzer/presidio_analyzer/recognizer_registry
recognizer_registry.py
recognizer_registry_provider.py
recognizers_loader_utils.py
Project Total

The report is truncated to 25 files out of 79. To see the full report, please visit the workflow summary page.

_{This report was generated by python-coverage-comment-action}

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (4)

presidio-analyzer/presidio_analyzer/recognizer_registry/recognizers_loader_utils.py:446

Same issue as the predefined branch: using key membership treats score_thresholds: None (often produced by config model dumping) as “present”, which can still clear a recognizer’s class-defined thresholds.

                has_score_thresholds = "score_thresholds" in recognizer_conf

presidio-analyzer/presidio_analyzer/recognizer_registry/recognizers_loader_utils.py:410

has_score_thresholds = "score_thresholds" in recognizer_conf will still be true when config validation/model_dump includes score_thresholds: None for an omitted key, so the class default can still be overwritten to {}. Use a value-based check so omission (or null) doesn’t count as an override, while {} still clears thresholds.

This issue also appears on line 446 of the same file.

            has_score_thresholds = "score_thresholds" in recognizer_conf

presidio-analyzer/presidio_analyzer/recognizer_registry/recognizer_registry.py:347

has_score_thresholds = "score_thresholds" in recognizer_config treats an explicit None value as an override, which would still clear any class-defined thresholds after normalize_score_thresholds(None) -> {}. To match the “omission keeps defaults” rule, base has_score_thresholds on the popped value being non-None.

        recognizer_config = recognizer_dict.copy()
        has_score_thresholds = "score_thresholds" in recognizer_config
        score_thresholds = normalize_score_thresholds(
            recognizer_config.pop("score_thresholds", None)
        )

presidio-analyzer/tests/test_recognizer_registry.py:266

These regression tests exercise RecognizerListLoader.get(...) directly, but the primary production path goes through RecognizerRegistryProvider + ConfigurationValidator, which can reintroduce score_thresholds: None for omitted keys via model_dump() and mask this bug. Consider adding a test which constructs a RecognizerRegistryProvider(registry_configuration=...) with the key omitted and asserts the class-defined thresholds are preserved end-to-end.

def _load_predefined(conf):
    return RecognizerListLoader.get(
        recognizers=[conf],
        global_regex_flags=re.DOTALL | re.MULTILINE | re.IGNORECASE,
        supported_languages=["en"],
    )

Copilot AI lite review requested due to automatic review settings August 4, 2026 08:42

Copilot started reviewing on behalf of omri374 August 4, 2026 08:43 View session

Update CHANGELOG with recent fixes and enhancements

c9bedde

Fixed various issues with recognizer configurations and behavior, including score thresholds, region codes, and language model parameters.

omri374 marked this pull request as ready for review August 4, 2026 08:45

Copilot AI reviewed Aug 4, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 4, 2026 08:45

Copilot started reviewing on behalf of omri374 August 4, 2026 08:46 View session

omri374 requested review from RonShakutai and SharonHart August 4, 2026 08:47

Copilot AI reviewed Aug 4, 2026

View reviewed changes

omri374 enabled auto-merge (squash) August 4, 2026 09:35

Merge branch 'main' into fix/score-thresholds-yaml-omission

8bf1a99

Copilot AI review requested due to automatic review settings August 5, 2026 11:29

Copilot started reviewing on behalf of omri374 August 5, 2026 11:30 View session

Copilot AI reviewed Aug 5, 2026

View reviewed changes

Navigation Menu

Uh oh!

fix(analyzer): keep recognizer score thresholds when config omits them - #2210

fix(analyzer): keep recognizer score thresholds when config omits them#2210
omri374 wants to merge 3 commits into
mainfrom
fix/score-thresholds-yaml-omission

omri374 commented Aug 4, 2026

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Aug 4, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

omri374 commented Aug 4, 2026

Change Description

Fix

Impact

Tests

Issue reference

Checklist

Uh oh!

github-actions Bot commented Aug 4, 2026

Coverage report (presidio-structured)

Uh oh!

github-actions Bot commented Aug 4, 2026

Coverage report (presidio-cli)

Uh oh!

github-actions Bot commented Aug 4, 2026

Coverage report (presidio-anonymizer)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

github-actions Bot commented Aug 4, 2026

Coverage report (presidio-image-redactor)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report (presidio-analyzer)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Aug 4, 2026 •

edited

Loading