Skip to content

README.md

Latest commit

 

History

History
219 lines (163 loc) · 8.84 KB

README.md

File metadata and controls

219 lines (163 loc) · 8.84 KB
Raw

FairTutor-Router

Python 3.11+ License Status

FairTutor-Router is the code and evaluation package for FairTutor: Equity-Aware Pedagogical LLM Routing for Budget-Constrained AI Tutoring (KDD 2026 AI for Education Day). It compares low-cost, premium, and evaluator-guided routing systems on a synthetic tutoring benchmark, with metrics for pedagogical quality, cost, access-tier AI Education Advantage Gap, escalation rate, and answer-leakage risk.

Overview

Premium AI tutors can provide clearer explanations and stronger scaffolding, but relying on premium models for every student query is expensive. FairTutor-Router tests a practical alternative: begin with a low-cost tutoring path, evaluate the response with a pedagogical rubric, repair borderline responses with a critic-rewriter, and escalate only when quality thresholds are not met.

The project is intentionally small and reproducible. It is a research prototype, not a production tutoring service.

Key Features

  • Five routing systems: low-cost only, premium only, naive difficulty routing, generic cascade routing, and FairTutor.
  • Pedagogy-aware evaluation: scores correctness, conceptual clarity, scaffolding, grade appropriateness, answer-leakage avoidance, empathy, and safety.
  • Evaluator-guided routing: uses fixed thresholds from configs/router.yaml, including stricter handling for hard queries and scaffold-sensitive cases.
  • Provider abstraction: model roles are configured in configs/models.yaml; code calls roles such as cheap, premium, and evaluator.
  • JSONL experiment traces: every response records model-call logs, token counts, estimated cost, evaluator scores, and escalation path.
  • Paper artifact scripts: tables, serving-cost estimates, confidence intervals, robustness checks, and threshold-sweep plots are regenerated from saved outputs.

Architecture

flowchart LR
    A[Student Query] --> B[Query Analyzer]
    B --> C[Pedagogical Planner]
    C --> D[Low-Cost Tutor]
    D --> E[Pedagogical Evaluator]
    E -->|Accept| F[Final Response]
    E -->|Borderline| G[Critic Rewriter]
    G --> E
    E -->|Fail| H[Premium Tutor]
    H --> I[Final Evaluation]
    I --> F
    F --> J[JSONL Logs and Metrics]
Loading

Routing Systems

System Description
cheap_only Always uses the configured low-cost tutor, then runs the common final evaluator for comparison.
premium_only Always uses the configured premium tutor; this is the quality target baseline.
naive_difficulty_router Routes easy and medium queries to the low-cost tutor, hard queries to premium.
generic_cascade_router Tries the low-cost tutor first and escalates if evaluator quality or correctness falls below threshold.
fairtutor_router Runs analyzer, planner, low-cost tutor, evaluator, optional critic rewrite, and selective premium escalation.

Repository Structure

fairtutor-router/
├── configs/                 # Model roles, routing thresholds, evaluator weights
├── data/                    # TutorAccessEval seed/generated JSONL datasets
├── experiments/             # Dataset generation, runs, aggregation, case studies
├── prompts/                 # Prompt templates loaded at runtime
├── results/                 # Curated public result artifacts
│   └── paper_main/          # Canonical 50-query paper run
├── scripts/                 # Paper table, CI, robustness, and Pareto scripts
├── src/fairtutor_router/    # Python package
└── tests/                   # Offline unit tests

Installation

git clone https://github.com/qyxu1994/fairtutor-router.git
cd fairtutor-router
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .env

Edit .env with credentials and model names for the providers you plan to use. .env is ignored by git; do not commit API keys.

The canonical paper results use saved outputs under results/paper_main/ and do not require new API calls to inspect or aggregate.

Configuration

Three YAML files control the experiment:

  • configs/models.yaml: provider and model for cheap, medium, premium, and evaluator.
  • configs/router.yaml: thresholds for FairTutor, generic cascade, and naive routing.
  • configs/eval.yaml: fixed evaluator weights and JSON-repair behavior.

Model names, API keys, base URLs, and token-cost rates should stay in .env or YAML config. Do not hardcode them in Python.

Running Experiments

Run one system:

python experiments/run_system.py \
  --system cheap_only \
  --dataset data/tutor_access_eval_seed.jsonl \
  --limit 10 \
  --out results/debug_run/cheap_only.jsonl

Run all five systems on the same dataset:

python experiments/run_all.py \
  --dataset data/tutor_access_eval_seed.jsonl \
  --limit 30 \
  --out_dir results/full_run

Aggregate a run and export case studies:

python experiments/aggregate_results.py \
  --results_dir results/full_run \
  --out_csv results/full_run/summary.csv \
  --out_md results/full_run/summary.md

python experiments/export_case_studies.py \
  --results_dir results/full_run \
  --dataset data/tutor_access_eval_seed.jsonl \
  --out results/full_run/case_studies.md \
  --top_k 8

Paper Artifacts

The canonical paper numbers are in results/paper_main/. These scripts read saved responses only unless otherwise noted.

# Table 1 standard-error column, scaffolding-by-difficulty breakdown, latency stats
python scripts/compute_paper_stats.py

# Table 2 serving-cost, cost-reduction, and premium-use columns.
# Serving cost is a relative token-weighted proxy, not a dollar-cost field in models.yaml.
python scripts/compute_serving_costs.py \
  --results_dir results/paper_main \
  --out_csv results/paper_main/serving_costs.csv

# Appendix B paired bootstrap confidence intervals
python scripts/bootstrap_ci.py --results_dir results/paper_main

The independent-judge robustness check and threshold-sensitivity frontier need additional passes:

# Re-score saved responses with the configured DeepSeek judge
python scripts/reeval_with_deepseek.py        # -> results/paper_main/reeval_deepseek/

# Materialize FairTutor candidates and replay the threshold sweep
python scripts/materialize_fairtutor_candidates.py   # -> results/paper_main/threshold_ablation/
python scripts/sweep_thresholds.py
python scripts/plot_pareto.py

Example Results

Canonical 50-query paper run:

System Quality Scaffolding AIED gap Within 0.5 of premium Serving cost (% premium) Premium use
premium_only 4.874 4.640 0.000 100% 100.0% 100%
fairtutor_router 4.760 4.380 0.114 92% 28.4% 18%
generic_cascade_router 4.695 4.160 0.179 88% 32.0% 28%
naive_difficulty_router 4.543 3.980 0.331 72% 39.7% 34%
cheap_only 4.400 3.640 0.474 78% 0.8% 0%

These are preliminary synthetic-benchmark results. They should be read as directional evidence, not as claims about classroom learning outcomes.

Dataset

data/tutor_access_eval_seed.jsonl contains hand-curated seed queries across math, reading, writing, science, language learning, and general tutoring. data/tutor_access_eval_generated.jsonl and data/tutor_access_eval_combined.jsonl contain generated benchmark variants used for larger runs.

To generate a new synthetic set with the configured generator model:

python experiments/generate_dataset.py \
  --num_per_bucket 12 \
  --out data/tutor_access_eval_generated.jsonl

Review generated data before using it in a paper or public benchmark.

Tests

pytest
pytest tests/test_router.py -v

The unit tests are offline and do not require API keys.

Safety and Limitations

  • FairTutor-Router is a research framework, not a replacement for teachers or human tutoring.
  • The benchmark is synthetic and single-turn; it does not measure real student learning gains.
  • Scores come from an LLM-as-judge rubric, which can introduce evaluator bias.
  • The current study focuses on access-tier equity across model cost tiers, not demographic equity.
  • Token prices and model quality change over time; keep model choices and costs configurable.

Citation

If you use this repository, code, dataset, or results, please cite:

@inproceedings{xu2026fairtutor,
  author    = {Xu, Qingyang},
  title     = {{FairTutor}: Equity-Aware Pedagogical {LLM} Routing for Budget-Constrained {AI} Tutoring},
  booktitle = {AI for Education Day at KDD 2026},
  year      = {2026},
  address   = {Jeju, Korea},
  url       = {https://github.com/qyxu1994/fairtutor-router}
}

License

This project is released under the MIT License. See LICENSE.