rusty-llm-jury

A Rust based CLI tool for estimating success rates when using LLM judges for evaluation.

---

Table of Contents

• Overview
• Installation
• Quick Start
• How It Works
• CLI Reference
• Examples
• Building from Source
• Testing
• Contributing
• License

---

Overview

When using Large Language Models (LLMs) as judges to evaluate other models or systems, the judge's own biases and errors can significantly impact the reliability of the evaluation. rusty-llm-jury provides a command-line tool to estimate the true success rate of your system by correcting for LLM judge bias using bootstrap confidence intervals.

Installation

From crates.io (when published)

cargo install llm-jury

From source

git clone https://github.com/udapy/rusty-llm-jury.git
cd rusty-llm-jury
cargo install --path .

Quick Start

Basic Estimation

# Estimate true success rate with bias correction
llm-jury estimate \
  --test-labels "1,1,0,0,1,0,1,0" \
  --test-preds "1,0,0,1,1,0,1,0" \
  --unlabeled-preds "1,1,0,1,0,1,0,1" \
  --bootstrap-iterations 20000 \
  --confidence-level 0.95

# Output:
# Estimated true pass rate: 0.625
# 95% Confidence interval: [0.234, 0.891]

Using Files

# Load data from CSV files
llm-jury estimate \
  --test-labels-file test_labels.csv \
  --test-preds-file test_preds.csv \
  --unlabeled-preds-file unlabeled_preds.csv

Synthetic Experiments

# Run TPR/TNR sensitivity analysis
llm-jury synth-experiment \
  --true-failure-rate 0.1 \
  --tpr-range 0.5,0.95 \
  --tnr-range 0.5,0.95 \
  --n-points 10 \
  --output results.json

How It Works

The tool implements a bias correction method based on the following steps:

1. Judge Accuracy Estimation: Calculate the LLM judge's True Positive Rate (TPR) and True Negative Rate (TNR) using labeled test data
2. Correction: Apply the Rogan-Gladen correction formula to account for judge bias:

   θ̂ = (p_obs + TNR - 1) / (TPR + TNR - 1)
   

   where p_obs is the observed pass rate from the judge
3. Bootstrap Confidence Intervals: Use bootstrap resampling to quantify uncertainty in the estimate

CLI Reference

llm-jury estimate

Estimate true pass rate with bias correction and confidence intervals.

Options:

• --test-labels <VALUES>: Comma-separated 0/1 values (human labels on test set)
• --test-preds <VALUES>: Comma-separated 0/1 values (judge predictions on test set)
• --unlabeled-preds <VALUES>: Comma-separated 0/1 values (judge predictions on unlabeled data)
• --test-labels-file <FILE>: Load test labels from CSV file
• --test-preds-file <FILE>: Load test predictions from CSV file
• --unlabeled-preds-file <FILE>: Load unlabeled predictions from CSV file
• --bootstrap-iterations <N>: Number of bootstrap iterations (default: 20000)
• --confidence-level <LEVEL>: Confidence level between 0 and 1 (default: 0.95)
• --output <FILE>: Save results to JSON file
• --format <FORMAT>: Output format: text, json, csv (default: text)

llm-jury synth-experiment

Run synthetic sensitivity experiments.

Options:

• --true-failure-rate <RATE>: True failure rate in unlabeled data (default: 0.1)
• --tpr-range <MIN,MAX>: TPR range to test (default: 0.5,1.0)
• --tnr-range <MIN,MAX>: TNR range to test (default: 0.5,1.0)
• --n-points <N>: Number of points in each range (default: 10)
• --n-test-positive <N>: Number of positive test examples (default: 100)
• --n-test-negative <N>: Number of negative test examples (default: 100)
• --n-unlabeled <N>: Number of unlabeled samples (default: 1000)
• --bootstrap-iterations <N>: Bootstrap iterations (default: 2000)
• --seed <SEED>: Random seed for reproducibility
• --output <FILE>: Output file (JSON or CSV based on extension)

Examples

Real-World Usage Pattern

# Step 1: Collect your data
echo "1,0,1,1,0,0,1,0" > test_labels.csv      # Human evaluation
echo "1,0,0,1,1,0,1,0" > test_preds.csv       # LLM judge on same data
echo "1,1,0,1,0,1,0,1,1,0" > unlabeled.csv    # LLM judge on target data

# Step 2: Estimate true success rate
llm-jury estimate \
  --test-labels-file test_labels.csv \
  --test-preds-file test_preds.csv \
  --unlabeled-preds-file unlabeled.csv \
  --format json \
  --output results.json

# Step 3: View results
cat results.json

Sensitivity Analysis

# Analyze how estimation varies with judge accuracy
llm-jury synth-experiment \
  --true-failure-rate 0.2 \
  --tpr-range 0.6,0.95 \
  --tnr-range 0.6,0.95 \
  --n-points 15 \
  --seed 42 \
  --output sensitivity_analysis.json

Building from Source

Prerequisites

• Rust 1.70+ (2021 edition)
• Cargo

Build Commands

# Clone repository
git clone https://github.com/ai-evals-course/rusty-llm-jury.git
cd rusty-llm-jury

# Build release version
make build

# Or using cargo directly
cargo build --release

# The binary will be at target/release/llm-jury

Development

# Format code
make fmt

# Run lints
make clippy

# Run tests
make test

# All checks
make check

Testing

Run the test suite:

cargo test

Run with coverage (requires cargo-tarpaulin):

cargo install cargo-tarpaulin
cargo tarpaulin --out html

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Inspired by Python judgy package, I am learning during my AI evals course with shreya and hamel.
• The Rogan-Gladen correction method for bias correction in diagnostic tests
• Bootstrap methodology for confidence interval estimation
• The Rust ecosystem for excellent tooling and libraries

---

Note: This tool assumes that your LLM judge performs better than random chance (TPR + TNR > 1). If your judge's accuracy is too low, the correction method may not be applicable.