Implement Knowledge Distillation

[Thamirawaran]

Goal

Transfer knowledge from a large, complex teacher model to a smaller, efficient student model.

Explanation of tasks

• 1. Propose a distillation methodology
• 2. Design a good prompt
• 3. Define output format of teacher
• 4. Prepare dataset for student training
• 5. Train the student model
• 6. Evaluate the student model

Propose a distillation methodology

• 1. Teacher model
• 2. Student model : Qwen2.5 VL 2B
• 3. Type of knowledge
• 4. Training method: offline
• 5. Type of knowledge transfer
• 6. Dataset: CarDD dataset

metrics

1. Exact match
2. F1 score

[MalithaPrabhashana]

VLM Distillation Methodology

Objective: Transfer car damage analysis knowledge from a large teacher VLM to a smaller student model (Qwen2.5 VL 2B) using the CarDD dataset.

Methodology:

1. Teacher Model Output Generation

• Process CarDD dataset images with the teacher model.
• Use prompt: "Analyze the car damage in this image. Provide the location, type, and severity of each damage. Also, generate an insurance report summarizing the damage and estimating the repair cost."
• Output format (JSON):

{
    "predictions": [
        {"location": "front bumper", "damage_type": "dent", "severity": "major"},
        {"location": "front fender", "damage_type": "dent", "severity": "major"},
        {"location": "front tire", "damage_type": "flat tire", "severity": "major"}
    ],
    "report": "Insurance Report: The vehicle sustained significant damage, including major dents on the front bumper and front fender, as well as a flat tire. Estimated repair cost: $1,800."
}

2. Dataset Preparation

• Create training pairs:
  (image, prompt) → teacher-generated JSON
• Use for supervised training of the student model.

3. Student Model Training

• Train Qwen2.5 VL 2B to generate the same JSON output.
• Input: Image + proper prompt.
• Treat as a text generation task, where the student outputs the JSON string.

4. Preventing Overfitting

• Split CarDD into training and validation sets.
• Monitor validation performance during training.

5. Evaluation

• Compare student outputs to teacher outputs (Using Metrices and Manual Evaluation)

[Thamirawaran]

Yeah this is accurate as we explained last time. I have few doubts on our methodology
We need to consider them

1. What teacher model is best? is it qwn72B VL or qwen32B vl or any other options and why they are good
2. Is it okay to fine tune model with json format? Won't it make any issues?
3. If it could cause any issues, what in which format we can fine tune model?
4. What are the metrics we are going to use? we need to test the accuracy in terms of teacher model.

[MalithaPrabhashana]

1. Since we can use openrouter, we can go with Qwen2.5 VL 72B Instruct (free) model.
   As we dicussed, if there is a limitation of the API key. we can use few API keys, and once limit of one key is over, we can move to the test API key. We can modify our script to support that feature.

2. It won't give any errors while fine tuning with JSON format caption. I tested in a colab note book, there the response was given according to the provided JSON format.