Skip to content

README.md

Latest commit

 

History

History
94 lines (67 loc) · 3.34 KB

README.md

File metadata and controls

94 lines (67 loc) · 3.34 KB
Raw

📊 Predictive Benchmark Runner

This directory contains the LLM-based evaluation runner. It consumes ground-truth group files (generated by the data-prep pipeline), prompts an LLM to rank solutions, and produces per-task and cross-task grading reports.

🧐 Functionality

  • 📥 Loads ground-truth comparison groups (e.g., groups_<task>_n2.json).
  • 📝 Constructs Prompts using task descriptions and data analysis reports.
  • 🧠 Queries LLMs to predict the better solution (pairwise) or rank multiple solutions.
  • 📈 Computes Metrics: Accuracy (for n=2), Spearman Correlation & Precision@k (for n>2).
  • 💾 Saves Reports including full interaction logs and machine-readable JSONs.

Entrypoint: run_bench.py

🏃 Quick Start

python -m skip_bench.grade.run_bench \
  --task-file /path/to/task_name.txt \
  --solutions-dir /path/to/solutions_root \
  --n 2 \
  --model deepseek-v3.2-thinking \
  --temperature 1.0 \
  --parallel 64 \
  --task-parallel 8 \
  --prompt-boost \
  --desc-dir /path/to/description \
  --da-dir /path/to/data_analysis/result \
  --cot

📂 Expected Inputs

  1. 📋 Task List (--task-file)

    • A text file with one task name per line.

  2. 🗂️ Solutions Root (--solutions-dir)

    • Structure: <solutions_dir>/<task>/ground_truth/groups_<task>_n*.json

  3. 📚 Task Resources (Optional but Recommended)

    • Mode 1 (Root): --tasks-root containing description/ and data_analysis/.
    • Mode 2 (Explicit): --desc-dir for descriptions and --da-dir for analysis.
    • Mode 3 (Raw): --raw-data-sample for raw data snippets.

  4. 🎯 Ground Truth Override

    • --groundtruth-file: Point to a specific JSON file (supports {task} placeholder).

⚙️ Key Arguments

Category Flag Description
Input --task-file Path to task list.
--solutions-dir Root for solutions & ground truth.
--n Group size (e.g., 2 for pairwise). 0 for all.
Model --model Model name (e.g., gpt-4).
--temperature Sampling temperature.
Execution --parallel Concurrency per task.
--task-parallel Number of tasks to run in parallel.
Prompting --prompt-boost Enhances instruction emphasis.
--cot Enaables Chain-of-Thought reasoning.
Output --partial-json Save checkpoints (default: on).

📤 Outputs

Artifacts are written to <solutions_dir>/<task>/report/:

  • 📄 Human-Readable Report: grade_report_<task>_... .txt (Metrics + Logs)
  • 🤖 Alignment JSON: alignment_<task>_... .json (Structured predictions)
  • 💾 Partial Checkpoint: grade_results_partial.json
  • 📊 Global Report: <solutions_dir>/report/grade_report_alltasks_*.txt

Notes

  • Group files are created by the data-prep pipeline (see the README in prepare_bench_subset).
  • If you pass --groundtruth-file, you must set --n to a non-zero value.
  • When using relative paths in group_entry.paths, keep --solutions-dir set so the runner can resolve files correctly.
  • The command examples above may not list all tunable flags. For a complete reference, run each script with --help or inspect the corresponding Python file and its argparse comments.