Install and fine-tune models with LLaMA Factory
LLaMA Factory is an open-source framework that simplifies the process of training and fine tuning large language models. It offers a unified interface for a variety of cutting edge methods such as SFT, RLHF, and QLoRA techniques. It also supports a wide range of LLM architectures such as LLaMA, Mistral and Qwen. This playbook demonstrates how to fine-tune large language models using LLaMA Factory CLI on your NVIDIA Spark device.
You'll set up LLaMA Factory on NVIDIA Spark with Blackwell architecture to fine-tune large language models using LoRA, QLoRA, and full fine-tuning methods. This enables efficient model adaptation for specialized domains while leveraging hardware-specific optimizations.
- Basic Python knowledge for editing config files and troubleshooting
- Command line usage for running shell commands and managing environments
- Familiarity with PyTorch and Hugging Face Transformers ecosystem
- GPU environment setup including CUDA/cuDNN installation and VRAM management
- Fine-tuning concepts: understanding tradeoffs between LoRA, QLoRA, and full fine-tuning
- Dataset preparation: formatting text data into JSON structure for instruction tuning
- Resource management: adjusting batch size and memory settings for GPU constraints
-
NVIDIA Spark device with Blackwell architecture
-
CUDA 12.9 or newer version installed:
nvcc --version -
Docker installed and configured for GPU access:
docker run --gpus all nvidia/cuda:12.9-devel nvidia-smi -
Git installed:
git --version -
Python environment with pip:
python --version && pip --version -
Sufficient storage space (>50GB for models and checkpoints):
df -h -
Internet connection for downloading models from Hugging Face Hub
-
Official LLaMA Factory repository: https://github.com/hiyouga/LLaMA-Factory
-
NVIDIA PyTorch container: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch
-
Example training configuration:
examples/train_lora/llama3_lora_sft.yaml(from repository) -
Documentation: https://llamafactory.readthedocs.io/en/latest/getting_started/data_preparation.html
- Duration: 30-60 minutes for initial setup, 1-7 hours for training depending on model size and dataset.
- Risks: Model downloads require significant bandwidth and storage. Training may consume substantial GPU memory and require parameter tuning for hardware constraints.
- Rollback: Remove Docker containers and cloned repositories. Training checkpoints are saved locally and can be deleted to reclaim storage space.
Check that your NVIDIA Spark system has the required components installed and accessible.
nvcc --version
docker --version
nvidia-smi
python --version
git --versionStart the NVIDIA PyTorch container with GPU access and mount your workspace directory.
Note
This NVIDIA PyTorch container supports CUDA 13
docker run --gpus all --ipc=host --ulimit memlock=-1 -it --ulimit stack=67108864 --rm -v "$PWD":/workspace nvcr.io/nvidia/pytorch:25.09-py3 bashDownload the LLaMA Factory source code from the official repository.
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-FactoryInstall the package in editable mode with metrics support for training evaluation.
pip install -e ".[metrics]"PyTorch is pre-installed with CUDA support.
To verify installation:
python -c "import torch; print(f'PyTorch: {torch.__version__}, CUDA: {torch.cuda.is_available()}')"Examine the provided LoRA fine-tuning configuration for Llama-3.
cat examples/train_lora/llama3_lora_sft.yamlNote
Login to your hugging face hub to download the model if the model is gated.
Execute the training process using the pre-configured LoRA setup.
huggingface-cli login # if the model is gated
llamafactory-cli train examples/train_lora/llama3_lora_sft.yamlExample output:
***** train metrics *****
epoch = 3.0
total_flos = 22851591GF
train_loss = 0.9113
train_runtime = 0:22:21.99
train_samples_per_second = 2.437
train_steps_per_second = 0.306
Figure saved at: saves/llama3-8b/lora/sft/training_loss.pngVerify that training completed successfully and checkpoints were saved.
ls -la saves/llama3-8b/lora/sft/Expected output should show:
- Final checkpoint directory (
checkpoint-21or similar) - Model configuration files (
config.json,adapter_config.json) - Training metrics showing decreasing loss values
- Training loss plot saved as PNG file
Test your fine-tuned model with custom prompts:
llamafactory-cli chat examples/inference/llama3_lora_sft.yaml
## Type: "Hello, how can you help me today?"
## Expect: Response showing fine-tuned behaviorllamafactory-cli export examples/merge_lora/llama3_lora_sft.yamlWarning
This will delete all training progress and checkpoints.
To remove all generated files and free up storage space:
cd /workspace
rm -rf LLaMA-Factory/
docker system prune -fTo rollback Docker container changes:
exit # Exit container
docker container prune -f| Symptom | Cause | Fix |
|---|---|---|
| CUDA out of memory during training | Batch size too large for GPU VRAM | Reduce per_device_train_batch_size or increase gradient_accumulation_steps |
| Cannot access gated repo for URL | Certain HuggingFace models have restricted access | Regenerate your HuggingFace token; and request access to the gated model on your web browser |
| Model download fails or is slow | Network connectivity or Hugging Face Hub issues | Check internet connection, try using HF_HUB_OFFLINE=1 for cached models |
| Training loss not decreasing | Learning rate too high/low or insufficient data | Adjust learning_rate parameter or check dataset quality |
Note
DGX Spark uses a Unified Memory Architecture (UMA), which enables dynamic memory sharing between the GPU and CPU. With many applications still updating to take advantage of UMA, you may encounter memory issues even when within the memory capacity of DGX Spark. If that happens, manually flush the buffer cache with:
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'