Skip to content

README.md

Latest commit

 

History

History
91 lines (62 loc) · 2.87 KB

README.md

File metadata and controls

91 lines (62 loc) · 2.87 KB
Raw

For more details how my model is work: check Intelligent..../pdf file...

Intelligent Tutor Response Evaluation System

Hybrid DistilBERT + Symbolic AI System for Automatic Tutor Feedback Classification

Achieved 90.5% accuracy (from 76% baseline) by combining DistilBERT, SymPy symbolic math, and domain-specific rules with LIME/SHAP explainability.

Problem

Automatically classify tutor responses into three categories:

  • 0 - Correct
  • 1 - Mistake
  • 2 - Unclear

Dataset consists of 4 tracks (math, tax/finance, logic, vague responses) — 1980 training and 496 test samples. Pure ML models fail on symbolic math and domain logic.

Solution Overview

Developed an iterative hybrid AI system that combines:

  • Rule-based symbolic reasoning (SymPy) for math & arithmetic validation
  • DistilBERT (fine-tuned transformer) for semantic understanding
  • Custom domain rules for tax, logic, and hedging detection
  • LIME & SHAP for local & global explainability

Key Innovation: One unified rule engine that covers all 4 tracks with high precision.

Key Features & Improvements

  • Started with TF-IDF + Logistic Regression → 76.0% accuracy
  • Added SymPy + basic rules → 82.1% accuracy (+6.1%)
  • Integrated DistilBERT + hybrid rules → 87.3% accuracy (+5.2%)
  • Final system with improved parsing + explainability → 90.5% accuracy (+14.5% overall)
  • F1-score for minority "Unclear" class improved by 720% (0.11 → 0.82)
  • Achieved 100% accuracy on all critical math test cases

Architecture

  • Input: Combined tutor + question + response text
  • Stage 1: Rule Engine (SymPy math solver + tax/logic/unclear rules)
  • Stage 2: DistilBERT (10 epochs, batch inference)
  • Stage 3: LIME (local) + SHAP (global) explanations
  • Output: Final label (0/1/2) + confidence + explanation

Results

Model Accuracy F1(Correct) F1(Mistake) F1(Unclear)
Baseline (TF-IDF + LR) 76.0% 0.10 0.86 0.11
SymPy + Rules 82.1% 0.68 0.89 0.25
Hybrid + DistilBERT 87.3% 0.80 0.91 0.52
Final System 90.5% 0.88 0.95 0.82

Tech Stack

  • Core: Python, PyTorch, Hugging Face Transformers
  • Model: DistilBERT (fine-tuned)
  • Symbolic AI: SymPy
  • Explainability: LIME, SHAP
  • Data: Pandas, NumPy
  • NLP: NLTK, TF-IDF (baseline)
  • Visualization: Matplotlib, Seaborn

How to Run

# Clone repo
git clone <your-repo-url>
cd tutor-evaluator

# Install dependencies
pip install -r requirements.txt

# Run inference
python inference.py --input "Solve 2y + 3 = 11" --response "y = 4"