Hierarchical evaluation improvements - Presidio Evaluator v-next (#172)

Author: RonShakutai

.github/workflows/ci.yml

@@ -13,7 +13,7 @@ jobs:
       contents: read
     strategy:
       matrix:
-        python-version: ['3.10', '3.11', '3.12', '3.13']
+        python-version: ['3.10', '3.11', '3.12', '3.13', '3.14']
 
     steps:
     - uses: actions/checkout@v4
@@ -32,33 +32,30 @@ jobs:
       uses: actions/setup-python@v5
       with:
         python-version: ${{ matrix.python-version }}
-    
-    - name: Cache Poetry dependencies
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v5
+
+    - name: Cache uv dependencies
       uses: actions/cache@v4
       with:
-        path: |
-          ~/.cache/pypoetry
-          ~/.cache/pip
-        key: ${{ runner.os }}-poetry-${{ matrix.python-version }}-${{ hashFiles('**/poetry.lock') }}
+        path: ~/.cache/uv
+        key: ${{ runner.os }}-uv-${{ matrix.python-version }}-${{ hashFiles('**/uv.lock') }}
         restore-keys: |
-          ${{ runner.os }}-poetry-${{ matrix.python-version }}-
+          ${{ runner.os }}-uv-${{ matrix.python-version }}-
     
     - name: Install dependencies
       run: |
-        python -m pip install --upgrade pip
-        pip install poetry
-        poetry install --with dev,ner
-        poetry run python -m spacy download en_core_web_sm
-        poetry run python -m spacy download en_core_web_lg
+        uv venv --seed
+        uv sync --extra dev
+        uv run python -m spacy download en_core_web_sm
+        uv run python -m spacy download en_core_web_lg
     
     - name: Run pytest
       run: |
-        poetry run pytest --runslow
+        uv run pytest --runslow
     
     - name: Clean up after tests
       if: always()
       run: |
-        # Clean up pip cache and poetry cache to free space for next matrix job
-        pip cache purge || true
-        poetry cache clear pypi --all -n || true
         df -h

.gitignore

@@ -191,3 +191,11 @@ datasets/
 *.spacy
 *.pickle
 /poetry.lock
+
+
+*.xml
+*.json
+*.html
+*.svg
+!synth_dataset_v2.json
+test_us002.py

.pre-commit-config.yaml

@@ -0,0 +1,23 @@
+repos:
+  - repo: local
+    hooks:
+      - id: ruff-format
+        name: ruff format
+        language: system
+        entry: uv run ruff format --check
+        types: [python]
+        pass_filenames: false
+
+      - id: ruff-check
+        name: ruff check
+        language: system
+        entry: uv run ruff check
+        types: [python]
+        pass_filenames: false
+
+      - id: pytest
+        name: pytest (unit tests only)
+        language: system
+        entry: uv run pytest -m "not slow and not integration" --ignore=tests/integration -q
+        pass_filenames: false
+        always_run: true

AGENTS.md

@@ -0,0 +1,53 @@
+# AGENTS.md — Codebase Patterns
+
+## Project Setup
+- Python managed with `uv` — always use `uv run` for Python, never bare `python` or `pip`
+- Virtualenv at `.venv/`
+- Quality checks: `uv run ruff check --fix`, `uv run ruff format`, `uv run pytest`
+- Fast entity-mapping tests: `pytest tests/entity_mapping/`
+
+## CanonicalMapper
+- Single-phase (Identify-only) — no projection phase
+- `analyze(results_df, min_severity='WARNING')` — COLLISION_SAME_BRANCH (INFO) hidden unless `min_severity='INFO'`
+- `get_mapped_results_dataframe()` returns `MappedResults` (frozen dataclass with `.original`, `.binary`, `.branch`, `.detailed`)
+- `get_mapping()` returns `{label: resolved}` dict — UNRESOLVED labels excluded
+- `get_issues()` filters by `_min_severity` — call after `analyze()` or `map()`
+
+## Five Issue Types (IssueType enum)
+- UNRESOLVED (ERROR, blocking)
+- COLLISION_CROSS_BRANCH (WARNING, blocking) — only raised when cross-branch co-occurrences outnumber same-branch ones for the prediction label
+- PREDICTION_ONLY (WARNING, blocking)
+- DATASET_ONLY (WARNING, non-blocking)
+- COLLISION_SAME_BRANCH (INFO, non-blocking)
+
+## _Resolution dataclass fields
+- `tier` — identification tier (EXACT, COUNTRY, COUNTRY_FALLBACK, FUZZY, UNRESOLVED)
+- `resolved` — resolved canonical entity name (or None for UNRESOLVED)
+- `score` — fuzzy match score (0.0–1.0, None for non-FUZZY tiers)
+- NO `canonical`, `projected`, or `projection_type` fields
+
+## IssueSeverity enum
+- Values are lowercase: `'error'`, `'warning'`, `'info'`
+- Use `.lower()` when converting from string; do NOT use `.upper()`
+
+## Hierarchical Evaluation
+- `calculate_hierarchical_scores(mapped_results: MappedResults)` returns `{"binary", "branch", "detailed"}`
+- NOT `{"L0", "L1", "L2"}` — use the string level names
+- `MappedResults` is in `presidio_evaluator/entity_mapping/data_objects.py`
+
+## level_helpers
+- `to_binary(label)` and `to_branch(label)` are now **instance methods on `EntityHierarchy`** (moved from the deleted `level_helpers.py`)
+- Call via `hierarchy_instance.to_binary(label)` / `hierarchy_instance.to_branch(label)`
+- `EntityHierarchy` has no imports from `mapper.py` — no circular deps
+- For branch lookups use `EntityHierarchy(canonical_depth=10)`
+
+## Notebooks
+- NB4: `4_Evaluate_Presidio_Analyzer.ipynb` — standard Presidio evaluation
+- NB5: `5_Evaluate_Custom_Presidio_Analyzer.ipynb` — custom model evaluation
+- NB6: `6_Interactive_Entity_Mapping.ipynb` — interactive mapping tutorial
+- All notebooks use `mapped_results = mapper.get_mapped_results_dataframe()` (not `mapped_df`)
+- `edit_notebook_file` requires VSC cell IDs (`#VSC-xxxx`) — use `copilot_getNotebookSummary` to get current IDs
+
+## Git
+- Pre-commit hooks: ruff-format, ruff-check, pytest (unit tests only)
+- Branch: `ralph/canonical-mapper-single-phase`

CHANGELOG.md

@@ -1,5 +1,40 @@
 # CHANGELOG
 
+## Unreleased
+
+> **Migration guide:** See [docs/migration-guide.md](docs/migration-guide.md) for step-by-step upgrade instructions.
+
+### Breaking Changes
+
+- **`evaluate_all()` removed** — raises `DeprecationError` at runtime. Replace with the three-step pipeline: `predict_dataset()` → `CanonicalMapper.get_mapped_results_dataframe()` → `calculate_score_on_df()`.
+- **`entity_mapping` parameter removed** from `SpanEvaluator`, `TokenEvaluator`, and `BaseEvaluator` — entity mapping is now the responsibility of `CanonicalMapper`.
+- **`compare_by_io` parameter removed** from evaluator constructors — BIO/BILUO prefix stripping is now performed by `CanonicalMapper`.
+- **`BaseEvaluator.from_dataset()` removed** — use `model.predict_dataset(dataset)` directly.
+- **Non-Presidio model wrappers removed**: `FlairModel`, `SpacyModel`, `StanzaModel`, `AzureAITextAnalyticsWrapper`. Add models directly through Presidio to evaluate them.
+- **Minimum Python version raised to 3.11** (was 3.10) — required by `numpy >= 2.4.0`.
+- **Package manager changed from Poetry to uv** — install with `uv sync`, run with `uv run`.
+
+### New Features
+
+- **`BaseModel.predict_dataset(dataset)`** — runs the model on a list of `InputSample` objects and returns a 5-column DataFrame (`sentence_id`, `token`, `annotation`, `prediction`, `start_indices`).
+- **`CanonicalMapper`** — replaces `EntityMappingHelper` with an improved four-tier auto-resolution strategy (`EXACT`, `COUNTRY`, `FUZZY`, `PENDING`). Key methods:
+  - `CanonicalMapper.from_dataset(dataset)` — builds a mapper from dataset labels.
+  - `mapper.get_mapped_results_dataframe(results_df)` — applies entity mapping to a predictions DataFrame.
+  - `mapper.get_mapping(mode='html' | 'text')` — returns the final `{raw_label: canonical | None}` dict.
+  - `mapper.map({"LABEL": "CANONICAL"})` — manually resolve pending labels.
+  - `mapper.render_html()` — display the resolution audit table in Jupyter.
+- **`TokenEvaluator.calculate_score_on_df(results_df)`** — score token-level predictions from a DataFrame.
+- **`SpanEvaluator.calculate_score_on_df(per_type, results_df)`** — score span-level predictions from a DataFrame.
+- **Ruff** — added as the project linter and formatter (`ruff.toml` at project root).
+- **Pre-commit hooks** — `ruff format`, `ruff check`, and `pytest` run automatically before every commit (`.pre-commit-config.yaml`).
+- **Test reorganisation** — tests are now grouped by topic (`tests/data_generator/`, `tests/entity_mapping/`, `tests/evaluation/`, `tests/models/`, `tests/integration/`). Integration tests are tagged with `pytest.mark.integration`.
+
+### Deprecations
+
+- **`evaluator.get_results_dataframe()`** — soft `DeprecationWarning` emitted at runtime. Replace with `model.predict_dataset(dataset)`.
+
+
+
 ## Version 0.2.5
 
 ### Improvements

NOTICE

@@ -351,7 +351,7 @@ tqdm
 
 `tqdm` is a product of collaborative work.
 Unless otherwise stated, all authors (see commit logs) retain copyright
-for their respective work, and release the work under the MIT licence
+for their respective work, and release the work under the MIT license
 (text below).
 
 Exceptions or notable authors are listed below
@@ -368,7 +368,7 @@ in reverse chronological order:
 [PR #96]: https://github.com/tqdm/tqdm/pull/96
 
 
-Mozilla Public Licence (MPL) v. 2.0 - Exhibit A
+Mozilla Public license (MPL) v. 2.0 - Exhibit A
 -----------------------------------------------
 
 This Source Code Form is subject to the terms of the

README.md

@@ -1,6 +1,6 @@
 # Presidio-research
 
-This package provides evaluation and data-science capabilities for 
+This package provides evaluation and data-science capabilities for
 [Presidio](https://github.com/microsoft/presidio) and PII detection models in general.
 
 It also includes a fake data generator that creates synthetic sentences based on templates and fake PII.
@@ -20,16 +20,13 @@ The easiest way to get started is by reviewing the notebooks.
 - [Notebook 3](notebooks/3_Split_by_pattern_number.ipynb): Provides tools to split the dataset into train/test/validation sets while avoiding leakage due to the same pattern appearing in multiple folds (only applicable for synthetically generated data).
 - [Notebook 4](notebooks/4_Evaluate_Presidio_Analyzer.ipynb): Shows how to use the evaluation tools to evaluate how well Presidio detects PII. Note that this is using the vanilla Presidio, and the results aren't very accurate.
 - [Notebook 5](notebooks/5_Evaluate_Custom_Presidio_Analyzer.ipynb): Shows how one can configure Presidio to detect PII much more accurately, and boost the f score in ~30%.
+- [Notebook 6](notebooks/6_Interactive_Entity_Mapping.ipynb): Explains the entity mapping process, which is crucial when evaluating multiple models each returning a different set of entities.
 
 ### Installation
 
->Note: Presidio evaluator requires Python version 3.9 or higher.
-
 #### From PyPI
 
 ``` sh
-conda create --name presidio python=3.12
-conda activate presidio
 pip install presidio-evaluator
 python -m spacy download en_core_web_sm # for tokenization
 python -m spacy download en_core_web_lg # for NER
@@ -43,31 +40,29 @@ To install the package:
 2. Install all dependencies:
 
 ``` sh
-# Install package+dependencies
-pip install poetry
-poetry install --with=dev
+# Install uv if not already installed
+pip install uv
 
-# Download tge spaCy pipeline used for tokenization
-poetry run python -m spacy download en_core_web_sm
+# Install package + dev dependencies
+uv sync --extra dev
 
-# To install with all additional NER dependencies (e.g. Flair, Stanza), run:
-# poetry install --with='ner,dev'
+# Download the spaCy pipeline used for tokenization
+uv run python -m spacy download en_core_web_sm
 
 # To use the default Presidio configuration, a spaCy model is required:
-poetry run python -m spacy download en_core_web_lg
+uv run python -m spacy download en_core_web_lg
 
 # Verify installation
-pytest
+uv run pytest
 ```
 
-Note that some dependencies (such as Flair and Stanza) are not automatically installed to reduce installation complexity.
+Note that some dependencies (such as Flair and Stanza) are no longer supported. Use Presidio Analyzer directly to add custom NER models.
 
 ## What's in this package?
 
 1. **Fake data generator** for PII recognizers and NER models
 2. **Data representation layer** for data generation, modeling and analysis
-3. Multiple **Model/Recognizer evaluation** files (e.g. for Presidio, Spacy, Flair, Azure AI Language)
-4. **Training and modeling code** for multiple models
+3. **Model/Recognizer evaluation** for Presidio Analyzer and custom Presidio recognizers
 5. Helper functions for **results analysis**
 
 ## 1. Data generation
@@ -120,13 +115,6 @@ The standardized structure, `List[InputSample]`, can be translated into differen
   InputSample.create_spacy_dataset(dataset, output_path="dataset.spacy")
   ```
 
-- Flair
-  ```python
-  from presidio_evaluator import InputSample
-  dataset = InputSample.read_dataset_json("data/synth_dataset_v2.json")
-  flair = InputSample.create_flair_dataset(dataset)
-  ```
-
 - json
   ```python
   from presidio_evaluator import InputSample
@@ -140,6 +128,7 @@ The presidio-evaluator framework allows you to evaluate Presidio as a system, a
 
 ## For more information
 
+- [Blog post on PII evaluation](https://omri-mendels.medium.com/evaluating-pii-detection-models-fa0c745d7a4c)
 - [Blog post on NLP approaches to data anonymization](https://towardsdatascience.com/nlp-approaches-to-data-anonymization-1fb5bde6b929)
 - [How to evaluate PII Detection output with Presidio Evaluator](https://tranguyen221.medium.com/how-to-evaluate-pii-detection-output-with-presidio-evaluator-3f2684ba3091)
 - [Conference talk about leveraging Presidio and utilizing NLP approaches for data anonymization](https://youtu.be/Tl773LANRwY)