This repository contains all necessary scripts and notebook(s) for fine-tuning the Unsloth Orpheus-3B language model on a high-quality Kazakh text dataset, kazakh-corpus2. 🚀 Project Summary Goal: Adapt a large language model (LLM) to the Kazakh language using efficient fine-tuning techniques (LoRA).
Base model: unsloth/orpheus-3b
Framework: Unsloth + Hugging Face transformers
Dataset: kazakh-corpus2
Training Strategy: Parameter-efficient fine-tuning with LoRA adapters.
⚙️ Fine-Tuning Configuration
| Parameter | Value |
|---|---|
r (LoRA rank) |
64 |
lora_alpha |
64 |
lora_dropout |
0.0 |
target_modules |
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
load_in_4bit |
False |
max_seq_length |
2048 |
num_train_epochs |
1 or more |
train/val/test split |
90% / 5% / 5% |
mixed_precision |
Enabled |
📦 Installation To install the required packages:
bash Copy Edit pip install -r requirements.txt If using Google Colab, the notebook handles the installation of extra dependencies like xformers, bitsandbytes, accelerate, etc.
📚 Dataset You can use your own preprocessed Kazakh dataset or refer to the publicly available kazakh-corpus2 (insert link if available). Preprocessing includes tokenization and splitting long sequences into 2048-token chunks.
🏁 Running the Notebook You can open the notebook in Colab or locally in Jupyter and run it cell by cell:
bash Copy Edit jupyter notebook orpheus_3B_fine_tuning_kz_vX.ipynb 📤 Pushing to Hugging Face Hub Use the Hugging Face CLI or transformers methods to push your trained model:
python Copy Edit model.push_to_hub("your-username/orpheus-3b-kazakh") tokenizer.push_to_hub("your-username/orpheus-3b-kazakh") 📜 License This project is licensed under the Apache 2.0 License, following the terms of the base model.
🤝 Acknowledgements
-
Unsloth team
-
Hugging Face
-
Contributors of issai/Kazakh_Speech_Corpus_2