Implement tiered PII detection pipeline

[AimelHassan]

Proposal for the detection module architecture based on the project brief.

Suggested three-tier approach:

1. Regex/pattern matching for structured PII (emails, IPs, phone numbers via phonenumbers library)
2. spaCy NER for names, organizations, locations
3. HuggingFace transformer fallback for context-dependent cases

The phonenumbers library by Google handles 200+ regional formats
which solves false positives from raw regex on phone numbers.

Detection tier selection could be configurable per policy:
real-time middleware prioritizes speed (tier 1+2),
batch analysis can use higher recall (all three tiers).

Happy to prototype this if mentors want a PoC.

[krittawee]

Hi! Just came across this while exploring the project for GSoC 2026. The three-tier architecture makes a lot of sense to me.
I had a question about Tier 3 — would the transformer always be invoked as a fallback, or is there a plan to only trigger it when the earlier tiers return low confidence? I'm not sure how spaCy exposes confidence scores so still learning, but wondering if that's a possible direction.

[Ushashi-Tech]

On the Tier 3 confidence threshold question — I'd suggest only invoking
the transformer when Tier 1 (regex) and Tier 2 (spaCy) both return
below a configurable confidence threshold (e.g. 0.7). This keeps
latency low for the common case while ensuring accuracy for ambiguous
inputs. The threshold could be exposed as a config option in policy.yaml.

[Ushashi-Tech]

On Tier 3 confidence threshold — I'd suggest only invoking the
transformer when Tier 1 (regex) and Tier 2 (spaCy) both return below
a configurable threshold (e.g. 0.7). This keeps latency low for the
common case. The threshold could be exposed as a config option in
policy.yaml so teams can tune it for their accuracy/speed tradeoff.

[AimelHassan]

On Tier 3 confidence threshold — I'd suggest only invoking the transformer when Tier 1 (regex) and Tier 2 (spaCy) both return below a configurable threshold (e.g. 0.7). This keeps latency low for the common case. The threshold could be exposed as a config option in policy.yaml so teams can tune it for their accuracy/speed tradeoff.

I agree with this architectural choice