Skip to content

Add a citation correctness scorer for RAG outputs #22

Description

@knisar

Split from #4 and being implemented in #27 by @M4h1m4. Follows the shape of #12.

A CitationCorrectnessScorer in rai_toolkit/scorers/ that checks whether citations in a response resolve to retrieved context that actually supports the cited claim. This catches two failures GroundednessScorer does not: a citation pointing at a source absent from the context, and a supported claim attributed to the wrong source. A response can be fully grounded and still cite incorrectly.

Conventions agreed on the #4 thread:

  • Cited spans are emitted verbatim and verified against the row before being trusted, reusing _verified_evidence_spans as-is (post-fix: accept normalized groundedness evidence spans #18 it handles whitespace/quote normalization with verbatim mapping back to row offsets).
  • Keep the existing {response_span, context_span} span shape and carry the citation marker alongside it.
  • Behavioral-refusal rows are skipped ahead of the empty-context branch, matching the FactualityJudge ordering and GroundednessScorer should skip behavioral-refusal rows like FactualityJudge #15.
  • Rows with no retrieved context return assessed=False, not a defaulted zero.
  • A response with no citations returns un-assessed with skipped: "no_citations". Citation presence enforcement is a separate policy metric, out of scope here.
  • Tests mock _call_judge, no key or network needed.

Retrieval relevance and context precision/recall stay on #4.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requeststatus: needs authorWaiting for the author to respond or revise

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions