Instructions to run LegalBench HELM Evaluation

[widmogrod]

Hi, as a noob, I was struggling with how to run this benchmark.

I found one way of running part of it via HELM, and I share the instructions below for everyone who needs some guidance.

If you're interested in putting those instruction into README.md I'm more than happy to create PR.

LegalBench HELM Evaluation

Run LegalBench evaluations using Stanford HELM framework.

Prerequisites

• uv installed
• OpenAI and Anthropic API keys

Quick Start (Evaluation Only)

# Set API keys
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."

# Create environment and install
uv venv
source .venv/bin/activate 
uv pip install 'crfm-helm[all]'

# Define models and tasks
# Full list of modes is available here https://github.com/stanford-crfm/helm/blob/main/src/helm/config/model_deployments.yaml
MODELS=(
  "openai/gpt-4o-mini-2024-07-18"
  "anthropic/claude-3-5-haiku-20241022"
)

TASKS=(
  "abercrombie"
  "corporate_lobbying"
  "international_citizenship_questions"
  "function_of_decision_section"
  "proa"
)

# Generate run entries
RUN_ENTRIES=""
for task in "${TASKS[@]}"; do
  for model in "${MODELS[@]}"; do
    RUN_ENTRIES+="legalbench:subset=${task},model=${model} "
  done
done

# Run evaluation
helm-run \
  --run-entries $RUN_ENTRIES \
  --suite my-legalbench-eval \
  --max-eval-instances 10 \
  --output-path ./results

# Summarize results
helm-summarize --suite my-legalbench-eval --output-path ./results

# View results in web interface
helm-server --suite my-legalbench-eval --output-path ./results

Open http://localhost:8000 to view interactive results.