Rebuild-LLM

Building a large language model from scratch, with notebooks for learning and a modular training package for repeatable experiments.

Quickstart (Modular Package)

python -m venv .venv
source .venv/bin/activate
pip install -e .
rebuild-llm-train-tiny \
  --text-file /absolute/or/relative/path/to/your_corpus.txt \
  --tokenizer word \
  --context-length 64 \
  --emb-dim 128 \
  --n-heads 4 \
  --n-layers 4 \
  --epochs 3

Project Layout

• src/rebuild_llm/config.py: typed experiment configs
• src/rebuild_llm/data/: tokenization + sequence datasets
• src/rebuild_llm/model/: attention, transformer blocks, GPT model
• src/rebuild_llm/engine/: training/evaluation + generation
• src/rebuild_llm/cli/train_tiny.py: runnable tiny experiment CLI
• Process/: notebook-based learning and experimentation workflow

Training Data Input

• Provide the dataset with --text-file <path>.
• Current pipeline expects UTF-8 plain text (typically .txt files).
• The text is read, tokenized, and then used for next-token training.
• Tokenizer choices:
  • --tokenizer char (default): character-level tokenizer
  • --tokenizer word: Process-style regex/word tokenizer with <|unk|> support

Extension Points

• Add a tokenizer (e.g., BPE/tiktoken) in src/rebuild_llm/data/tokenizer.py
• Add new datasets/objectives in src/rebuild_llm/data/ and src/rebuild_llm/engine/
• Add model variants (RoPE, RMSNorm, MoE, etc.) in src/rebuild_llm/model/
• Add extra CLI commands for pretrain, classify, instruct-finetune in src/rebuild_llm/cli/