Skip to content

Add AI-generated text detection benchmarks to Evaluation and Benchmarks - #414

Open
chajahmed wants to merge 1 commit into
keon:mainfrom
chajahmed:add-detection-benchmark
Open

Add AI-generated text detection benchmarks to Evaluation and Benchmarks#414
chajahmed wants to merge 1 commit into
keon:mainfrom
chajahmed:add-detection-benchmark

Conversation

@chajahmed

Copy link
Copy Markdown

This matters because of NLP evaluation/classification: AI-vs-human text detection is a binary text-classification task, and detector robustness against adversarial rewrites (paraphrasers, humanizer tools) is an active evaluation problem without a dedicated place on this list yet.

Adds a new "AI-generated text detection:" grouping under Evaluation and Benchmarks with two entries:

  • RAID (ACL 2024) - the largest adversarial benchmark for AI-text detectors, already well-established in the literature.
  • AI2Human Detection Benchmark - a smaller labeled corpus (raw-AI / human / paraphrase-tool-humanized / LLM-humanizer-humanized) plus a script to score samples against a live detector API. Disclosure: I maintain the AI2Human detector/humanizer product this corpus benchmarks, included alongside RAID as a second reference point rather than the sole entry.

Happy to adjust placement, trim to just RAID, or drop the AI2Human entry entirely if it reads as too self-promotional for the list's bar.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant