Skip to content

README_en.md

Latest commit

 

History

History
276 lines (207 loc) · 11.6 KB

README_en.md

File metadata and controls

276 lines (207 loc) · 11.6 KB
Raw

ChartQA with Qwen3-VL: Knowledge Distillation & QLoRA Fine-tuning

English | 中文

Leveraging Qwen3-VL-32B's reasoning capability to construct an augmented dataset via knowledge distillation, then fine-tuning Qwen3-VL-2B with QLoRA to achieve near-large-model chart numerical reasoning under single-GPU deployable conditions.

Project Structure

ChartQA/
├── sft/                    # Supervised Fine-Tuning
│   ├── __init__.py
│   ├── config.py           # Hyperparameter configuration (dataclass system)
│   ├── model.py            # Model loading and LoRA adaptation
│   ├── data_loader.py      # Dataset loading and preprocessing
│   ├── collator.py         # Data collation and label masking
│   ├── callbacks.py        # Training callbacks (monitoring, early stopping)
│   ├── trainer.py          # Trainer setup
│   └── train.py            # SFT training entry point
│
├── ChartQADataset/         # ChartQA dataset
│   ├── train/              # Training split
│   ├── val/                # Validation split
│   └── test/               # Test split
│
├── test_model.py           # Evaluation script (six metrics)
├── requirements.txt        # Python dependencies
├── README_en.md            # This document
└── LICENSE                 # MIT License

Installation

pip install -r requirements.txt

Project Background

Why Use a 2B Small Model?

Dimension Qwen3-VL-32B-Thinking Qwen3-VL-8B-Thinking Qwen3-VL-2B-Thinking Gap
FP16 VRAM (weights only) ~64 GB ~16 GB ~4 GB 16x (32B vs 2B)
Minimum Deployment Hardware Multiple A100 / H100 Single 5090 32GB Single RTX 4060 8GB 10-30x hardware cost difference
Single Query Latency ~2-5 seconds ~0.8-1.5 seconds ~0.3-0.8 seconds 3-6x

Three Core Contradictions

Contradiction 1: Deployment Cost vs Reasoning Ability
  → 32B performs well but is too expensive; 2B is affordable but lacks numerical reasoning

Contradiction 2: Data Annotation vs Annotation Cost
  → To teach 2B reasoning chains, we need (question, reasoning_chain, answer) triplets,
    but manually writing tens of thousands of structured reasoning chains is prohibitive

Contradiction 3: End-to-End SFT vs Reasoning Ability
  → If we only use (question, answer) pairs for SFT, the small model degenerates into
    a "guessing" mode — seeing "2019" and outputting the tallest bar's value,
    rather than truly learning to read axes, extract values, and compute

Solution: Use 32B large model's reasoning output to generate training annotations for the 2B small model, teaching it to "think step-by-step like a large model" and achieve near-large-model numerical reasoning accuracy under single-GPU deployable model size.

Dataset Construction: Knowledge Distillation

Original ChartQA Dataset

The original ChartQA dataset contains only (image, question, label) three fields, without reasoning chain annotations:

{
  "image": "multi_col_20134.png",
  "question": "What is the estimated cyber security market size of the defense and intelligence sector in 2017?",
  "label": "250"
}

Distillation Pipeline

Using Qwen3-VL-32B-Thinking as the teacher model, we perform item-by-item reasoning on the ChartQA dataset to automatically generate high-quality reasoning chains. The teacher model's output format is:

Inpy prompt template:

You are a helpful assistant capable of visual reasoning for answering questions about charts. 
Given a chart image and a question, you will first analyze the chart step by step, then provide the final answer in a JSON format:
{answer: "your_answer"}
Make sure to include a detailed reasoning process before the final answer.
Here is the chart image: {image_path}
Question: {question}

Output format:

<think>reasoning content</think>
<answer>answer</answer>

Example:

input:

"""
You are a helpful assistant capable of visual reasoning for answering questions about charts. 
Given a chart image and a question, you will first analyze the chart step by step, then provide the final answer in a JSON format:
{answer: "your_answer"}
Make sure to include a detailed reasoning process before the final answer.
Here is the chart image: multi_col_20134.png
Question: What is the estimated cyber security market size of the defense and intelligence sector in 2017?
"""
<think>Got it, let's look at the bar chart. The question is about the defense and intelligence sector's cyber security market size in 2017. The chart has two segments for each year: blue is "Other Public Sector" and black is "Defence and Intelligence".

For 2017, the black segment (defence and intelligence) shows a value. Let me check the numbers. The black part for 2017 is labeled as 250 million GBP. Let me confirm. The y-axis is value in million GBP. The black bar in 2017 reaches up to 250. Yes, the label on top of the black segment for 2017 is 250.</think>
<answer>250</answer>

Data Conversion

The distilled _augmented.json is parsed and split into _converted.json (5-field format) to adapt to the SFT training code:

{
  "image": "multi_col_20134.png",
  "question": " What is the estimated cyber security market size of the defense and intelligence sector in 2017?",
  "think": "Got it, let's look at the bar chart. The question is about the defense and intelligence sector's cyber security market size in 2017. The chart has two segments for each year: blue is \"Other Public Sector\" and black is \"Defence and Intelligence\".\nFor 2017, the black segment (defence and intelligence) shows a value. Let me check the numbers. The black part for 2017 is labeled as 250 million GBP. Let me confirm. The y-axis is value in million GBP. The black bar in 2017 reaches up to 250. Yes, the label on top of the black segment for 2017 is 250.",
  "answer": "250",
  "label": "250"
}

Key Design Decisions:

  • think field: The step-by-step reasoning chain generated by the teacher model, serving as a supervision signal to teach the model "analyze chart structure → locate data → compute → give answer"
  • answer field: The teacher model's output answer (not directly used during training)
  • label field: The ground truth from the original dataset; during training, the model's target answer uses the real label rather than the teacher model's output, ensuring dual supervision signals where the reasoning chain is a reference and the answer is the standard

Dataset Structure

ChartQADataset/
├── train/
│   ├── train_augmented.json     # Raw distillation output (with <think> tags)
│   ├── train_converted.json     # Converted training data (5-field format)
│   └── png/                     # Chart images
├── val/
│   ├── val_augmented.json
│   ├── val_converted.json
│   └── png/
└── test/
    ├── test_augmented.json
    └── png/

QLoRA Fine-tuning

Model Architecture

  • Base Model: Qwen3-VL-2B-Thinking (vision-language model with CoT reasoning capability)
  • Fine-tuning Method: QLoRA (Quantized Low-Rank Adaptation)
  • Trainable Parameters: ~1-2% (only training attention layer projection matrices)

LoRA Principle

LoRA injects trainable rank decomposition matrices into the pre-trained model's weight matrices through low-rank decomposition, freezing the original model parameters and only training the low-rank adapters:

W = W₀ + BA

Where W₀ is the frozen pre-trained weight, and B and A are trainable low-rank matrices (r << d).

QLoRA further quantizes the base model to 4-bit, significantly reducing VRAM usage while maintaining accuracy.

LoRA Configuration

Parameter Value Description
r (rank) 16 Rank of low-rank matrices, controls trainable parameters
lora_alpha 32 Scaling factor, typically set to 2r
target_modules q_proj, k_proj, v_proj, o_proj 4 projection matrices in attention layers
lora_dropout 0.05 Prevents overfitting
bias none Do not train bias terms
task_type CAUSAL_LM Causal language modeling task

Training Configuration

Parameter Value
Learning rate 2e-5
Batch size 2 (per device)
Gradient accumulation 8 steps
Effective batch size 16
Max sequence length 1024
Precision BF16
Optimizer adamw_torch
NEFTune noise alpha 5.0 (increases input robustness)
Early Stopping patience 3

Training Data Flow

Raw JSON → Load data → Build conversation → Apply template → Processor encoding → Label masking → Training batch
  1. Load data: Read 5-field format data from train_converted.json
  2. Build conversation: Combine (system_prompt, question, think, label) into a three-turn dialogue
  3. Apply template: Convert dialogue to full text sequence using apply_chat_template
  4. Processor encoding: Images go through the visual encoder, text through the tokenizer
  5. Label masking: Use _mask_labels to mask the part before <think> as -100, training only the model to generate reasoning chains and answers
  6. Training batch: Dynamically pad to batch maximum length

Training Data Format

Training sample labels are formatted into a unified template:

{think_text} {answer: "label"}

The system prompt explicitly requires the model to "perform step-by-step reasoning, then output the final answer in JSON format: {answer: 'your_answer'}".

Training Entry Point

cd sft
python train.py

Evaluation System

Six Quantitative Evaluation Metrics

Metric Description
Exact Match Normalized prediction matches ground truth exactly
Numeric Accuracy Tolerance-based numerical comparison (relative error < 0.1%)
Token F1 Token-level F1 score between prediction and ground truth
Format Compliance Percentage of outputs containing {answer: '...'} JSON format
Answer Extraction Rate Percentage of successfully extracted JSON answers from model outputs
Avg Response Length Average token count of model outputs

Evaluation Script

python test_model.py

Results are saved as JSON files for easy cross-model version comparison.

Distilled 2B vs Original 8B Comparison

The core comparison in this project is between the distilled + QLoRA fine-tuned 2B model and the original un-fine-tuned 8B base model:

Metric Qwen3-VL-2B (Distilled + QLoRA SFT) Qwen3-VL-8B-Thinking (Base) Description
Exact Match 90.1% (64/71) 90.1% (64/71) Tied
Numeric Accuracy 93.0% (53/57) 93.0% (53/57) Comparable numerical reasoning
Format Compliance 100% (71/71) 100% (71/71) Both consistently output JSON
Answer Extraction Rate 100% (71/71) 98.6% (70/71) 2B has more stable formatting
Mean Token F1 0.901 0.901 Identical answer token overlap

Key Findings:

  • The distilled + QLoRA fine-tuned 2B model ties with the original 8B model in Exact Match (90.1% vs 90.1%)
  • Numerical reasoning accuracy is identical (93.0% vs 93.0%), and Token F1 is exactly the same (0.901)
  • Yet the 2B model's VRAM requirement is only 1/4 of 8B's, reducing deployment cost by an order of magnitude
  • The 2B model's inference latency is 2-3x lower than 8B's, making it more suitable for high-concurrency scenarios
  • Significant cost-performance advantage: Using 1/4 of the VRAM and 1/3 of the inference time to achieve the same reasoning accuracy as the 8B base model