Add initial Inspect AI evaluation harness

[thomas-bartlett]

Summary

Adds an initial Inspect AI evaluation harness under evaluations/codeguard-inspect/.

This first version uses password storage as the initial secure-coding task and compares baseline model output with CodeGuard-guided output. It establishes the project structure, task wiring, deterministic scoring pattern, and local validation flow for future CodeGuard evaluation work.

What This Adds

• Isolated uv subproject for optional Inspect AI evaluations
• One initial secure-coding task with baseline and CodeGuard-guided variants
• Deterministic scorer for generated output
• Focused pytest coverage for task wiring and scorer behavior
• README instructions for installation, running, validation, and current limitations

Notes

This PR is intended as a seed harness, not a full benchmark or effectiveness claim. Future updates can add additional tasks, scoring approaches, CodeGuard delivery modes, agent-based runs, benchmark adapters, or execution-based checks where appropriate.

Validation

• uv run pytest
• uv run python -m compileall codeguard_harness
• uv run inspect list tasks codeguard_harness/tasks/password_storage.py

Related

• Refs Add evaluation harness for Project CodeGuard #86
• Related to Discussion AI assisted Code Generation: Testing Harness Literature Review #70

[bact]

Suggested change

DISCUSSION_URL = "https://github.com/cosai-oasis/project-codeguard/discussions/70"

We probably don't need this but can keep for the purpose of being a seed.

[bact]

Suggested change

"discussion_url": DISCUSSION_URL,

We probably don't need this but can keep for the purpose of being a seed.

[bact]

Thank you. Looks solid for the job and we can build further on this.

Btw, to have this eval registered on Inspect Evals at https://ukgovernmentbeis.github.io/inspect_evals/ we need a small YAML file in the Inspect Eval Register. Something like this:

title: "CodeGuard Password Storage Eval"
description: |
  Evaluate password storage security practice
contributors:
  - thomas-bartlett
tags:                                      # optional; the upstream repo name is added automatically
  - CodeGuard
  - Coding
  - Password
  - Security
tasks:
  - name: password_storage                 # must match the @task function name
    task_path: evaluations/codeguard-inspect/codeguard_harness/tasks/password_storage.py
source:
  repository_url: https://github.com/cosai-oasis/project-codeguard
  repository_commit: <40-char SHA>         # grab from the commit page on GitHub; tags and branches aren't accepted
  comment: "Project CodeGuard is an open-source, model-agnostic security framework that embeds secure-by-default practices into AI coding agent workflows."

(use example from https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/register/example_eval.yaml )