Process Supervision for Math Finetuning

A controlled LoRA finetuning study on Qwen/Qwen2.5-Math-7B-Instruct asking a simple but important question: does supervising the reasoning process help math performance more than supervising only the final answer?

At a Glance

• Base model: Qwen/Qwen2.5-Math-7B-Instruct
• Finetuning method: LoRA
• Training data: GSM8K train
• Main evaluation: GSM8K test
• Core variants:
  • answer_only
  • reasoning_answer
  • formatted_reasoning_answer
• Main finding: process supervision improves GSM8K accuracy from 37.98% to 80.14%, and explicit step-by-step formatting gives a small additional gain to 80.52%

Research Question

Under the same base model and a similar training budget, does reasoning + answer supervision outperform answer-only supervision on math reasoning tasks? If process supervision already helps, does making the reasoning structure more explicit with Step 1 / Step 2 / ... / Final Answer help even more?

Main Results

Experiment	Accuracy	Structure / Reasoning Rate	Final Answer: Tag Rate	Interpretation
answer_only v1	37.98%	0.00%	100.00%	Learns answer formatting well, but reasoning performance remains weak
reasoning_answer v1	80.14%	100.00%	96.59%	Process supervision dramatically improves performance and reliably elicits reasoning
formatted_reasoning v1	80.52%	100.00% step-tag	95.91%	Structured step-by-step supervision gives a small additional gain and highly consistent step tags

Headline comparisons:

• reasoning_answer v1 vs answer_only v1: +42.16 accuracy points
• formatted_reasoning v1 vs reasoning_answer v1: +0.38 accuracy points

Figures

Accuracy Comparison

Output Behavior

Training Curves

What This Repo Shows

• Process supervision changes more than output style. In this controlled setup, it substantially improves mathematical reasoning accuracy.
• answer_only supervision is enough to teach the model to emit Final Answer: ..., but not enough to produce strong problem-solving behavior.
• Natural-language reasoning supervision already captures most of the gain.
• Further structuring the reasoning into explicit Step 1 / Step 2 / ... yields a small but real additional improvement and makes the reasoning format highly consistent.

Example Output Patterns

Successful reasoning_answer outputs often look like:

Reasoning:
...
Final Answer: 18

Successful formatted_reasoning_answer outputs often look like:

Step 1: ...
Step 2: ...
Final Answer: 18

This matters because the repo is not only measuring final accuracy. It also tracks whether the model actually learned to emit the intended reasoning structure.

Repository Layout

math_process_supervision_qwen/
  data/
    sft/
  reports/
    data/
    figures/
  src/
    data/   # preprocessing, SFT building, token-length inspection
    eval/   # GSM8K evaluation scripts and answer extraction
    train/  # LoRA training scripts
    visualize/  # plotting scripts for public-facing figures

Generated checkpoints, adapters, and evaluation JSON files are written to outputs/ during remote runs and are intentionally not versioned in Git.

Core Scripts

Data Preparation

• src/data/preprocess_gsm8k_variants.py
• src/data/build_sft_dataset_reasoning.py
• src/data/build_sft_dataset_answer_only.py
• src/data/build_sft_dataset_formatted_reasoning.py
• src/data/inspect_token_lengths.py

Training

• src/train/train_lora_reasoning_smoke.py
• src/train/train_lora_reasoning_v1.py
• src/train/train_lora_answer_only_v1.py
• src/train/train_lora_formatted_reasoning_v1.py

Evaluation

• src/eval/eval_gsm8k_reasoning_smoke.py
• src/eval/eval_gsm8k_reasoning_v1.py
• src/eval/eval_gsm8k_answer_only_v1.py
• src/eval/eval_gsm8k_formatted_reasoning_v1.py

Visualization

• src/visualize/plot_gsm8k_results.py
• src/visualize/plot_training_curves.py

Reproduce the Pipeline

1. Build training variants

python src/data/preprocess_gsm8k_variants.py
python src/data/build_sft_dataset_reasoning.py
python src/data/build_sft_dataset_answer_only.py
python src/data/build_sft_dataset_formatted_reasoning.py

2. Inspect sequence lengths

python src/data/inspect_token_lengths.py

3. Train a variant

Examples:

python src/train/train_lora_reasoning_v1.py
python src/train/train_lora_answer_only_v1.py
python src/train/train_lora_formatted_reasoning_v1.py

4. Evaluate on GSM8K

Examples:

python src/eval/eval_gsm8k_reasoning_v1.py
python src/eval/eval_gsm8k_answer_only_v1.py
python src/eval/eval_gsm8k_formatted_reasoning_v1.py

5. Generate public-facing figures

python src/visualize/plot_gsm8k_results.py

If trainer_state.json files have been synced back from remote outputs, you can also generate training curves:

python src/visualize/plot_training_curves.py path/to/trainer_state.json --title "Training Curve" --output-name training_curve.png

Current Status

Completed:

• GSM8K preprocessing
• SFT dataset construction
• Formal answer_only v1 training and evaluation
• Formal reasoning_answer v1 training and evaluation
• Formal formatted_reasoning v1 training and evaluation
• Three-way comparison on full GSM8K test

Planned next:

• MATH subset evaluation
• qualitative error analysis polishing
• final report and repository packaging

Report

For a more narrative write-up of the experiments, see:

• reports/experiment_summary.md