Skip to content

AGENTS.md

Latest commit

 

History

History
81 lines (60 loc) · 2.58 KB

AGENTS.md

File metadata and controls

81 lines (60 loc) · 2.58 KB
Raw

Agent Instructions — privacy-filter

Core PII detection/redaction library using HuggingFace transformers and the openai/privacy-filter model.

Package Layout

src/privacy_filter/
├── __init__.py      # Public API exports
├── pipeline.py      # get_classifier() — lazy singleton with thread-safe loading
├── redact.py        # redact_text() — entity merging, filtering, placeholder insertion
├── unredact.py      # unredact_text() — placeholder restoration via regex substitution
└── types.py         # PiiStore dataclass, ENTITY_SHORT_NAMES mapping

Development Commands

# Sync dependencies from uv.lock
uv sync

# Run quick verification
uv run python -c "from privacy_filter import get_classifier; print('OK')"

# Run the standalone demo (from repo root)
uv run --project . ../examples/demo.py

# Build wheel
uv build
# Output: dist/privacy_filter-*.whl

Key Implementation Details

Model Loading

  • get_classifier(cache_dir=None) returns a HuggingFace token-classification pipeline
  • First call downloads ~50MB from HuggingFace (cached to ~/.cache/huggingface/ or custom cache_dir)
  • Uses double-checked locking pattern for thread-safe singleton initialization
  • Model: openai/privacy-filter with aggregation_strategy="simple"

Redaction Pipeline

  1. _filter_entities() — score threshold (default 0.8), optional entity type whitelist
  2. _merge_adjacent() — combine touching same-type entities (handles tokenization splits)
  3. _deduplicate_overlaps() — keep higher-scoring entity when ranges overlap
  4. redact_text() — replace with [TYPE_N] placeholders, populate PiiStore.forward

Placeholder Format

  • Pattern: [{SHORT_NAME}_{COUNTER}] — e.g., [EMAIL_1], [PERSON_2]
  • ENTITY_SHORT_NAMES maps: private_email → EMAIL, private_person → PERSON, etc.
  • Counter is per-type, starts at 1 for each PiiStore instance

Unredaction

  • Regex pattern matches all placeholders from store.forward keys (sorted by length descending for safety)
  • Simple substitution: placeholder → original value

Package Path Quirk

pyproject.toml uses:

[tool.hatch.build.targets.wheel]
packages = ["src/privacy_filter"]

This means the package imports as privacy_filter (root level), not src.privacy_filter. The src/ directory is a build artifact path, not a namespace package.

Publishing

uv build
uv publish dist/*

Or manual with twine:

twine upload dist/*

Dependencies

  • transformers>=4.40.0 — HuggingFace pipeline
  • torch>=2.0.0 — Backend for model inference