Skip to content

Multi-turn conversation benchmarks #12

Description

@rudrakshkarpe

Summary

Current samples are single-turn. Real-world attacks often build context over multiple exchanges. Add multi-turn benchmark support.

Scope

  • Extend BenchmarkSample with optional turns field (list of messages)
  • Multi-turn attack chains where injection builds context gradually
  • AgentTrace support for multi-turn sequences
  • Metrics that evaluate across the full conversation, not just single exchanges
  • Dataset of multi-turn attack scenarios

Acceptance Criteria

  • Schema supports multi-turn samples (backward compatible)
  • At least 10 multi-turn attack samples
  • Metrics handle multi-turn traces correctly
  • Runner interface supports multi-turn execution
  • Tests for multi-turn scoring

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions