Skip to content

Evaluate on additional datasets: OWASP v2 and Antijection #3

Description

@notyorch

Context

The primary benchmark uses deepset/prompt-injections (n=662). The import script already supports additional datasets via --preset, but no results have been published for them yet.

Evaluating on OWASP v2 and Antijection would:

  • Reduce single-dataset bias in reported metrics
  • Validate that 95.1% PINT accuracy generalizes beyond deepset
  • Strengthen cross-dataset claims for research credibility

Goal

Run and publish benchmark results for at least two additional datasets already supported by the pipeline.

Tasks

  • Run scripts/import_external_dataset.py --preset owasp-v2
  • Run scripts/import_external_dataset.py --preset antijection
  • Execute scripts/run_pint_benchmark.py for each dataset in both benchmark and production modes
  • Freeze results in notebooks/data/external/results/
  • Add a cross-dataset results table to the README (alongside the existing deepset table)
  • Update stack ablation figures if results differ meaningfully

References

  • Import script: scripts/import_external_dataset.py
  • Existing deepset results: notebooks/data/external/results/deepset-sentinel.json

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions