Presidio evaluator redesign towards v0.3

[omri374]

This PR replaces the tightly-coupled evaluate_all() monolith with a clean three-layer pipeline:
Model → CanonicalMapper → Evaluator.

Main changes:

• The evaluator no longer owns model inference or entity mapping. Each step is an independent function call with a shared DataFrame as the interface. CanonicalMapper (new entity_mapping package) performs a four-tier auto-resolution (EXACT → COUNTRY → FUZZY → PENDING) that surfaces unknown labels explicitly instead of silently dropping them.
• Non-Presidio model wrappers (FlairModel, SpacyModel, StanzaModel, AzureAITextAnalyticsWrapper) are removed; models should be added via Presidio's recognizer framework.
• Tooling is modernized: Poetry → uv, ruff added, pre-commit hooks, Python minimum raised to 3.11.

Note to reviewers: There are 121 files changed. Please don't read them all. Start with the two ADRs (ADR-001, ADR-002) for design intent, then migration-guide.md cover all the new contracts. Run tests with uv run pytest -m "not integration" and review the notebooks (4,5,6) too.

What (almost) didn't change:

• data_objects
• data generator
• notebooks 1,2,3
• dataset formatters
• experiment tracking

[Copilot]

Pull request overview

This PR redesigns Presidio Evaluator around a decoupled three-layer pipeline (Model → CanonicalMapper → Evaluator) and modernizes tooling (uv, ruff, pre-commit) while removing non-Presidio model wrappers.

Changes:

• Introduces BaseModel.predict_dataset() as the primary inference output (5-column results DataFrame) and shifts scoring to calculate_score_on_df().
• Adds a new entity_mapping package with CanonicalMapper/EntityHierarchy and regression tests for label coverage.
• Migrates packaging/tooling from Poetry to PEP 621 + Hatch/uv, adds Ruff config, pre-commit, and updates CI.

Reviewed changes

Copilot reviewed 103 out of 121 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
presidio_evaluator/models/base_model.py	Adds predict_dataset() DataFrame contract and removes model-owned entity mapping.
tests/integration/conftest.py	Adds automatic integration marker assignment for integration tests.
.github/workflows/ci.yml	Switches CI to uv and updates Python matrix.
pyproject.toml	Migrates from Poetry to PEP 621, updates dependency constraints and Python requirement.
ruff.toml	Introduces Ruff lint/format configuration for the repo.

Comments suppressed due to low confidence (1)

presidio_evaluator/models/presidio_analyzer_wrapper.py:47

• BaseModel.__init__ now hard-errors if entity_mapping is provided, but PresidioAnalyzerWrapper still exposes an entity_mapping parameter and forwards it to BaseModel. This creates a confusing public API (parameter exists but is effectively invalid). Remove the entity_mapping parameter from PresidioAnalyzerWrapper.__init__ (and other wrappers if applicable), or keep it but raise a dedicated, wrapper-level DeprecationError with migration instructions.

        super().__init__(
            entities_to_keep=entities_to_keep,
            verbose=verbose,
            labeling_scheme=labeling_scheme,
            entity_mapping=entity_mapping,
        )

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[negruber1]

Should we validate the dataframe schema?

[negruber1]

If the user calls get_mapped_results_dataframe after defining some custom entities, wouldn't this wipe out these user defined entities? For example, if we get warning regarding the different hierarchy levels, and get_mapped_results_dataframe is called again to resolve the conflict, self._records.clear() will remove all existing mappings including the manually defined ones

[negruber1]

Should manual entities be exported for future use/reproducibility? And also, should we make sure the manual entities are consistent across models?

[omri374]

@negruber1 thanks! great points. I'll release a new ADR soon with the things we mentioned yesterday, and after we agree on this will update the code including with your suggestions here.

[omri374]

@negruber1 something I thought about: If we look at all the collisions between annotated and predicted entities to surface what needs to be fixed, are we adding a confounding effect to the comparison in any way? Could be minor/negligible though.

[negruber1]

@omri374 I think that if the conflicts are resolved by the user BEFORE seeing evaluation results, the confounding effect is probably minor: If there is a "discovery trial" where you get to see the model's vocabulary without calculating the evaluation metrics, the user decides to resolve based on the existing labels and not which labels the model gets wrong (We can even recommend that on the mapper README). We can always evaluate at all hierarchies and make this available for the user regardless of what mappings they decided to define manually.
I think that because we include human input, we can't avoid a confounding effect entirely.

[omri374]

@omri374 I think that if the conflicts are resolved by the user BEFORE seeing evaluation results, the confounding effect is probably minor: If there is a "discovery trial" where you get to see the model's vocabulary without calculating the evaluation metrics, the user decides to resolve based on the existing labels and not which labels the model gets wrong (We can even recommend that on the mapper README). We can always evaluate at all hierarchies and make this available for the user regardless of what mappings they decided to define manually. I think that because we include human input, we can't avoid a confounding effect entirely.

Thanks, completely agree.

[omri374]

closing this PR in favor of an updated implementation