nue

The homebrew-scale LLM 🐵🦝🐯🐍

Warning

Work in progress — nothing useful here yet.

I'd like to gain practical experience with transformers, particularly by understanding their architecture and real-world applications, with a focus on small-scale LLMs. To achieve this, I decided to create a tiny LLM. My goal is to integrate it into web applications, games, and iOS apps that interest me.

Goal

Build a small language model that generates grammatically correct sentences.

Philosophy

Keep it simple, but not simplistic.

Dataset

名前	ライセンス	備考
Wikimedia Wikipedia	CC BY-SA 3.0
livedoor ニュースコーパス	CC BY-ND 2.1 JP	データソースは llm-book/livedoor-news-corpus

Tokenizer

日英のコーパスを用いて SentencePiece + Unigram で学習します。

• byte_fallback=True で OOV (語彙外) 回避
• vocab_size は 32,000

(1) コーパスを生成

以下のコマンドを実行すると、 build/corpus.txt が生成されます。

$ poetry run nue build-corpus

(2) Tokenizer を学習

以下のコマンドを実行すると、 build/tokenizer.model と build/tokenizer.vocab が生成されます。

$ poetry run nue train-tokenizer

Training

poetry run nue train

License

Apache License 2.0

Name origin 🐵🦝🐯🐍

The name Nue (鵺, pronounced "noo-eh") originates from Japanese folklore, referring to a mythical creature with the head of a monkey, body of a tanuki (raccoon dog), legs of a tiger, and tail of a snake.

Inspired by this creature, this project serves as a personal exploration into building language models—combining newly-learned, diverse techniques into one cohesive yet eclectic model. Additionally, "Nue" can be read similarly to "new," symbolizing this as a new, homebrew-scale LLM crafted from scratch.

References

Excellent articles and papers that I've read:

• Language Models are Unsupervised Multitask Learners
• GPT in 60 Lines of NumPy | Jay Mody
• Reproducing GPT-2 (124M) in llm.c in 90 minutes for $20 · karpathy/llm.c · Discussion #481
• Training Compute-Optimal Large Language Models a.k.a. "Chinchilla Scaling Law"
• Physics of Language Models