Fine-tuning GPT-Neo on Custom Dataset

This project involves fine-tuning the GPT-Neo model on a custom dataset using the Hugging Face Transformers library.

Requirements

Python 3.6 or later

PyTorch 1.8.0 or later

Transformers 4.5.0 or later

You can install the required packages using pip:

pip install torch transformers

Dataset

The dataset should be a text file with one sample per line. The path to this file is specified in the file_path variable in the script.

Usage

Clone the repository:

git clone https://github.com/Nepomuk5665/Nvidia_chat_ai.git

Navigate to the cloned repository:

cd yourrepository

Run the script:

python train.py

The script will load the dataset, tokenize it, and fine-tune the GPT-Neo model on it. The trained model and tokenizer will be saved in the directory specified by the model_dir variable.

Output

The trained model and tokenizer are saved in the directory specified by the model_dir variable. You can use these for generating text or further fine-tuning.