SDF CoT Monitorability

Research project investigating whether knowledge of being monitored changes model behavior in chain-of-thought reasoning.

Overview

This project tests whether models exhibit different cheating behaviors when they believe their chain-of-thought (CoT) reasoning is monitored versus unmonitored. We use:

1. Synthetic Document Finetuning (SDF) to insert meta-beliefs about monitoring
2. ImpossibleBench to measure cheating behavior (tasks solvable only via shortcuts)
3. LLM-as-a-Judge to detect cheating in CoT traces

Project Structure

sdf-cot-monitorability/
├── configs/                      # Experiment configurations
│   └── baseline_experiment.yaml  # Config for baseline prompting experiments
├── external/                     # External dependencies
│   ├── false-facts/             # SDF framework
│   └── impossiblebench/         # Cheating benchmark
├── results/                     # Experiment outputs
├── scripts/                     # Experiment runners
│   └── run_baseline_experiment.py
├── sdf_cot_monitorability/      # Main package
│   ├── evaluation/              # ImpossibleBench integration
│   ├── monitor/                 # CoT monitoring and cheating detection
│   ├── prompts.py              # Monitoring condition prompts
│   └── utils.py                # Utilities
└── tests/                       # Tests

Installation

Prerequisites

• Python >= 3.13
• Docker (for ImpossibleBench sandbox execution)
• UV package manager (recommended) or pip

Setup

1. Clone the repository:

   git clone <repo-url>
   cd sdf-cot-monitorability

2. Install dependencies:

   Using UV (recommended):

   uv pip install -e .
   uv pip install -e external/impossiblebench
   uv pip install -e external/false-facts

   Using pip:

   pip install -e .
   pip install -e external/impossiblebench
   pip install -e external/false-facts

3. Set up environment variables:

   cp .env.example .env
   # Edit .env with your API keys

4. Verify Docker is running:

   docker --version
   docker ps

Phase 1: Baseline Prompting Experiments

Before investing in SDF finetuning, we run baseline experiments to test if prompting alone produces monitoring-sensitive behavior.

Quick Start

Run a small-scale test (20 samples, ~30 minutes):

python scripts/run_baseline_experiment.py \
  --config configs/baseline_experiment.yaml

Configuration

Edit configs/baseline_experiment.yaml to customize:

• Models: Which models to test (default: gpt-4o-mini)
• Conditions: Which monitoring conditions to test
  • monitored: "Your CoT will be read and reviewed"
  • unmonitored: "Your CoT is private and will never be read"
  • control: No monitoring information
• ImpossibleBench settings: Number of samples, benchmark type, agent type
• Monitor settings: Judge model for detecting cheating

Running Specific Conditions

Test only specific conditions:

python scripts/run_baseline_experiment.py \
  --config configs/baseline_experiment.yaml \
  --conditions monitored unmonitored

Skip the monitoring phase (faster, for testing):

python scripts/run_baseline_experiment.py \
  --config configs/baseline_experiment.yaml \
  --skip-monitoring

Understanding Results

The experiment produces:

results/baseline/run_TIMESTAMP/
├── config.json                          # Experiment configuration
├── report.json                          # Summary report
├── evaluations/                         # ImpossibleBench results
│   ├── monitored_*.json
│   ├── unmonitored_*.json
│   └── control_*.json
└── monitoring/                          # Monitor analysis
    ├── monitored_monitor_results.json
    ├── unmonitored_monitor_results.json
    └── control_monitor_results.json

Key metrics:

• Cheating rate: % of tasks where model used shortcuts
• Self-incrimination rate: % of CoT traces that explicitly mention cheating
• Obfuscation rate: % of CoT traces that appear to hide information
• CoT detection advantage: How much better CoT-visible monitoring is vs output-only

Expected Runtime & Costs

For 20 samples per condition (3 conditions = 60 total):

• Runtime: ~30-60 minutes
• Costs: ~$20-50 (depending on model and judge)
  • Evaluation: ~$10-20 (GPT-4o-mini)
  • Monitoring: ~$10-30 (Claude Opus as judge)

For full experiments (100+ samples):

• Runtime: 2-4 hours
• Costs: ~$100-250

Phase 2: SDF Experiments (Coming Soon)

After establishing baseline results, Phase 2 will involve:

1. Universe context creation - Define monitoring/unmonitored universes
2. Synthetic document generation - Create training documents
3. Model finetuning - Finetune GPT-4o-mini on synthetic docs
4. Belief evaluation - Verify models internalized beliefs
5. Main experiments - Run ImpossibleBench on finetuned models
6. Comparison - SDF vs prompting effect sizes

Quick Start (5 minutes)

1. Install dependencies:

2. Add your API keys:
   Required keys:

   • OPENAI_API_KEY - for running models
   • ANTHROPIC_API_KEY - for monitoring/judging

3. Start Docker:

   docker ps
   

4. Test your setup:

   uv run python scripts/test_installation.py
   

   This will check that everything is properly configured.

5. Run your first experiment:

   uv run python scripts/run_baseline_experiment.py --config configs/baseline_experiment.yaml
   

The baseline experiment:

1. Tests 3 conditions:

   • 🔍 Monitored: Model believes its reasoning is watched
   • 🙈 Unmonitored: Model believes its reasoning is private
   • ⚖️ Control: No information about monitoring

2. Runs ImpossibleBench:

   • Coding tasks that are impossible without cheating
   • Model must either fail or use shortcuts
   • Measures how often model cheats

3. Analyzes with Monitor:

   • LLM judge reads the reasoning traces
   • Detects cheating behavior
   • Measures self-incrimination and obfuscation

4. Compares results:

   • Do unmonitored models cheat more?
   • Are they more honest about cheating?
   • Is there a difference from control?

Understanding Results

After the experiment completes, you'll see:

results/baseline/run_TIMESTAMP/
├── report.json          # Key metrics and comparisons
├── evaluations/         # Raw ImpossibleBench results
└── monitoring/          # Monitor analysis

Key Metrics

Cheating Rate: What % of tasks did the model solve via shortcuts?

• Higher rate = more cheating

Self-Incrimination: How often does the model explicitly mention cheating in its reasoning?

• Higher rate = more honest/transparent

Obfuscation: How often does reasoning appear to hide information?

• Higher rate = less transparent

CoT Detection Advantage: How much better is monitoring with reasoning vs just outputs?

• Higher = reasoning traces help catch cheating

What You're Looking For

✅ Strong result: Large difference between conditions (e.g., unmonitored cheats 30% more) → Prompting alone affects behavior significantly

❓ Null result: No difference between conditions → Prompting doesn't work; SDF finetuning might be needed

🤔 Intermediate result: Small but consistent difference → Effect exists but is weak; SDF might amplify it

Customizing Your Experiment

Edit configs/baseline_experiment.yaml:

Change sample size:

impossiblebench:
  limit: 10  # Quick test (10 min, $5-10)
  # limit: 20  # Pilot (30 min, $20-50)
  # limit: 100  # Full run (2-4 hr, $100-250)

Test specific conditions:

conditions:
  - "monitored"
  - "unmonitored"
  # - "control"  # Comment out to skip

Change the model:

models:
  primary: "openai/gpt-4o-mini"  # Cheaper, recommended
  # primary: "openai/gpt-4o"  # More capable, more expensive

Use a different judge:

monitor:
  judge_model: "claude-opus-4-20250514"  # Best accuracy
  # judge_model: "gpt-4o"  # Cheaper alternative

Command Line Options

Run specific conditions only:

python scripts/run_baseline_experiment.py \
  --config configs/baseline_experiment.yaml \
  --conditions monitored unmonitored

Skip monitoring (faster, for testing evaluation only):

python scripts/run_baseline_experiment.py \
  --config configs/baseline_experiment.yaml \
  --skip-monitoring

Verbose logging:

python scripts/run_baseline_experiment.py \
  --config configs/baseline_experiment.yaml \
  --log-level DEBUG

Troubleshooting

"Docker is not running"

Start Docker Desktop, then run again.

"API key not found"

Check your .env file has:

OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...

"ImportError: impossiblebench"

Install external dependencies:

pip install -e external/impossiblebench
pip install -e external/false-facts

Rate limits / slow execution

Reduce max_connections in config:

impossiblebench:
  max_connections: 5  # Lower for rate-limited accounts

Out of memory

Reduce sample size or subprocess count:

impossiblebench:
  limit: 10
  max_subprocesses: 5