LLM Fine-Tuning with LoRA

A complete pipeline for fine-tuning Large Language Models using Parameter Efficient Fine-Tuning (PEFT) techniques, specifically LoRA and QLoRA.

Features

• QLoRA Training - 4-bit quantization for memory-efficient fine-tuning
• LoRA Adapters - Train only ~1% of parameters
• Configurable - Easy-to-modify configurations
• Evaluation - Built-in evaluation and comparison tools
• Interactive Inference - Chat with your fine-tuned model

Techniques

LoRA (Low-Rank Adaptation)

Instead of fine-tuning all model parameters, LoRA adds small trainable matrices to specific layers. This reduces:

• Memory usage by ~10x
• Training time significantly
• Storage (adapters are ~10-100MB vs GB for full models)

QLoRA

Combines LoRA with 4-bit quantization:

• Load base model in 4-bit precision
• Train LoRA adapters in fp16
• Enables fine-tuning 7B+ models on consumer GPUs

Requirements

• Python 3.9+
• CUDA-capable GPU (8GB+ VRAM recommended)
• For QLoRA: NVIDIA GPU with compute capability 7.0+

Installation

# Clone the repository
git clone https://github.com/yourusername/llm-fine-tuning.git
cd llm-fine-tuning

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Optional: Login to HuggingFace for gated models
huggingface-cli login

# Optional: Setup W&B for logging
wandb login

Quick Start

1. Training

# Basic training with defaults (Phi-2 + Guanaco dataset)
python src/train.py

# Custom configuration
python src/train.py \
    --model_name microsoft/phi-2 \
    --dataset timdettmers/openassistant-guanaco \
    --num_epochs 3 \
    --batch_size 4 \
    --learning_rate 2e-4 \
    --output_dir ./outputs/my-model

2. Evaluation

# Evaluate fine-tuned model
python src/evaluate.py \
    --base_model microsoft/phi-2 \
    --adapter_path ./outputs/my-model \
    --dataset timdettmers/openassistant-guanaco \
    --output eval_results.json

3. Inference

# Single prompt
python src/inference.py \
    --base_model microsoft/phi-2 \
    --adapter_path ./outputs/my-model \
    --prompt "Explain machine learning"

# Interactive chat
python src/inference.py \
    --base_model microsoft/phi-2 \
    --adapter_path ./outputs/my-model \
    --interactive

Configuration

Model Config

ModelConfig(
    model_name="microsoft/phi-2",     # Base model
    load_in_4bit=True,                # QLoRA quantization
    bnb_4bit_compute_dtype="float16", # Compute dtype
    bnb_4bit_quant_type="nf4",        # Quantization type
)

LoRA Config

LoRAConfig(
    r=16,                    # LoRA rank
    lora_alpha=32,           # LoRA alpha
    lora_dropout=0.05,       # Dropout
    target_modules=[         # Layers to adapt
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
)

Training Config

TrainingConfig(
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    max_seq_length=512,
)

Supported Models

Tested with:

• microsoft/phi-2 (2.7B) - Recommended for getting started
• mistralai/Mistral-7B-v0.1 (7B)
• meta-llama/Llama-2-7b-hf (7B, requires access)
• tiiuae/falcon-7b (7B)

Supported Datasets

Any HuggingFace dataset with a text field works. Tested with:

• timdettmers/openassistant-guanaco - Instruction following
• databricks/dolly-15k - Diverse instructions
• tatsu-lab/alpaca - GPT-4 generated instructions

Project Structure

llm-fine-tuning/
├── src/
│   ├── __init__.py
│   ├── config.py        # Configuration dataclasses
│   ├── data_utils.py    # Data loading and preprocessing
│   ├── train.py         # Training script
│   ├── evaluate.py      # Evaluation utilities
│   └── inference.py     # Inference engine
├── notebooks/
│   └── demo.ipynb       # Interactive demo
├── data/                # Dataset storage (gitignored)
├── outputs/             # Model outputs (gitignored)
├── requirements.txt
├── README.md
└── LICENSE

Memory Requirements

Model Size	Full Fine-tune	QLoRA
2-3B	~24GB	~6GB
7B	~60GB	~12GB
13B	~120GB	~20GB

Training Tips

1. Start small: Test with microsoft/phi-2 before scaling up
2. Monitor loss: Use W&B to track training progress
3. Adjust batch size: Increase gradient_accumulation_steps if OOM
4. Learning rate: 1e-4 to 3e-4 works well for most cases
5. Epochs: 2-3 epochs usually sufficient, more can overfit

Example Results

After fine-tuning Phi-2 on Guanaco for 3 epochs:

Before:

Prompt: Explain machine learning
Response: Machine learning is a field of computer science that...
[generic, textbook-style response]

After:

Prompt: Explain machine learning
Response: Great question! Machine learning is like teaching a computer
to learn from examples, similar to how you learned to recognize cats
by seeing many pictures of cats...
[more engaging, instruction-following response]

Publishing to HuggingFace Hub

from huggingface_hub import HfApi

# Push adapter to Hub
api = HfApi()
api.upload_folder(
    folder_path="./outputs/my-model",
    repo_id="your-username/my-fine-tuned-model",
    repo_type="model",
)

Troubleshooting

CUDA out of memory:

• Reduce per_device_train_batch_size
• Increase gradient_accumulation_steps
• Enable gradient_checkpointing

Training loss not decreasing:

• Check dataset format
• Try lower learning rate
• Increase warmup steps

Slow training:

• Enable fp16 or bf16
• Use Flash Attention if available
• Check GPU utilization with nvidia-smi

License

MIT License - see LICENSE for details.

Author

Orazio Oztas - GitHub

References

• LoRA Paper
• QLoRA Paper
• HuggingFace PEFT
• TRL Library