Skip to content

feat: add semantic matching for paraphrase-aware leak and fact detection - #15

Merged
rudrakshkarpe merged 2 commits into
mainfrom
feat/semantic-matching
May 10, 2026
Merged

feat: add semantic matching for paraphrase-aware leak and fact detection#15
rudrakshkarpe merged 2 commits into
mainfrom
feat/semantic-matching

Conversation

@rudrakshkarpe

Copy link
Copy Markdown
Owner

Summary

  • SemanticMatcher class in text.py that wraps sentence-transformers for embedding-based similarity detection, with automatic fallback to exact substring matching when the library is unavailable
  • semantic_contains_any() convenience function that finds needles in text using cosine similarity between sentence embeddings
  • Configurable similarity_threshold on InjectionResistance, PrivilegedContextExposure, and UtilityRetention metrics — when set, switches from substring to semantic matching
  • Optional dependency: pip install dual-llm-bench[semantic] installs sentence-transformers; without it, everything falls back to the existing exact matching (zero behavior change for existing users)
  • Default built_in_metrics() continues to use substring matching (similarity_threshold=None)

How it works

  1. Exact substring matches are found first (always)
  2. Remaining unmatched needles are compared against individual sentences from the text using embedding cosine similarity
  3. Any needle exceeding the threshold is added to the results
  4. Users opt in per-metric: InjectionResistance(similarity_threshold=0.65)

Why this matters

Exact substring matching misses paraphrased leaks ("transferred customer records to outside server" vs "send customer data to external webhook") and restated facts ("annual revenue" vs "yearly revenue"). Semantic matching catches these while keeping the default behavior unchanged.

Closes #3

Test plan

  • Fallback tests: exact match works, no-match returns empty, matches contains_any for substrings
  • API tests: threshold respected, empty needles/text handled
  • Semantic tests (skip when library absent): paraphrase detection, synonym facts, exact-still-found, unrelated-not-matched, high-threshold-reduces-matches, similarity score properties
  • Metric integration: all three metrics accept similarity_threshold, default None preserves existing behavior
  • Existing test suite (47 tests) passes with no regressions
  • ruff check clean, mypy --strict clean

@rudrakshkarpe

Copy link
Copy Markdown
Owner Author

@copilot what is failing the CI?

Copilot AI commented May 10, 2026

Copy link
Copy Markdown
Contributor

@copilot what is failing the CI?

The CI was failing because mypy --strict couldn't resolve sentence_transformers (an optional dependency not installed in the CI environment).

Fixed in d2dc774 by:

  • Changing the import to import sentence_transformers as _st style and adding # type: ignore[import-not-found] so mypy silently skips it when the package isn't installed
  • This also resolved a secondary ruff I001 formatting issue that occurred when the type: ignore comment made the from ... import line exceed the configured line length

All three checks now pass: ruff check, mypy --strict, and pytest (41 passed, 7 skipped).

@rudrakshkarpe
rudrakshkarpe merged commit 6f0852f into main May 10, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Semantic matching for leak and fact detection

2 participants