This repository is a first-principles developer guide for understanding language-model components and building a small decoder-only Transformer end to end. It starts with “what is AI?” and progresses through tokenization, attention, training, inference, alignment, multimodality, serving, continual learning, and experimental AI systems.
Scope: This is an educational systems guide with one runnable mini-transformer. It is not a production LLM framework, a paper-reproduction suite, or evidence that the advanced roadmap has been implemented.
- Python-first teaching code — Most examples use Python/NumPy/PyTorch, with a small number of C++/kernel sketches for systems context
- First-principles explanations — Derivations, shape traces, diagrams, and code are labelled by implementation status
- Broad systems view — Tokenization, training, inference, alignment, multimodality, serving, safety, and continual-learning concepts
- Practical dataset guides — Load pinned external datasets from Hugging Face, Kaggle, HTTPS files, and Common Crawl, or build a versioned text dataset from authorized raw sources
- Connected to Aarambh AI — The systems topics are informed by the aarambh-ai Rust project, while this repository uses Python for accessible experiments
- Separates implementation from proposals — Chapter 37 is explicitly an experimental Aarambh AI research roadmap, not a report of completed results
Every chapter now states what kind of material it contains:
| Label | Meaning |
|---|---|
| Core teaching chapter | Small executable examples that demonstrate the real operation on CPU |
| Advanced educational chapter | Research-grounded explanation plus simplified code or pseudocode; production details may be omitted |
| Systems simulation | Connects components and prints illustrative metrics but is not model training |
| Research proposal | An unvalidated architecture idea that requires experiments and ablations |
The cohesive runnable implementation is code/minillm.py. It includes a tokenizer, causal Transformer, real backward pass, AdamW updates, validation, checkpointing, loading, and autoregressive generation. Tests verify that optimizer steps change parameters and checkpoints round-trip.
make check # run manuscript validation and unit tests
make demo # run a small CPU-friendly training demonstration
make train # run a longer educational CPU training job
git clone https://github.com/AarambhDevHub/llm-from-scratch.git
cd llm-from-scratch
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
make install
make check
make demomake demo uses a deliberately small CPU configuration and one PyTorch CPU thread to avoid thread-pool overhead on tiny tensors. Use make train for a longer educational run. Generated checkpoints are written under checkpoints/ and ignored by Git.
| You | Start Here |
|---|---|
| Complete beginner to AI/ML | Chapter 1 |
| Developer who can code but never built a model | Chapter 1 |
| ML engineer who uses frameworks (PyTorch, etc.) but wants to understand the internals | Chapter 5 (or skim first 4) |
| Rust/Python/C++ developer interested in LLM systems | Start anywhere, each chapter is self-contained |
| Researcher wanting a full-system view | All chapters |
Prerequisites: Basic programming knowledge (any language). High school math (algebra).
PART 1: FOUNDATIONS
├── 01-what-is-ai.md — AI vs ML vs DL, history, the big picture
├── 02-how-computers-learn.md — Training, inference, data, models, parameters
├── 03-math-for-ml.md — Linear algebra, calculus, probability you actually need
├── 04-neural-networks-101.md — Perceptrons, forward pass, activation functions
PART 2: THE TRANSFORMER REVOLUTION
├── 05-text-to-numbers.md — Tokenization: BPE, WordPiece, SentencePiece
├── 06-embeddings.md — From tokens to meaning: word vectors, positional encoding
├── 07-attention.md — The secret sauce: QKV, multi-head, causal masking
├── 08-transformer.md — The full architecture: blocks, residuals, norms
├── 09-feedforward-moe.md — FFN, SwiGLU, Mixture of Experts, routing
PART 3: TRAINING
├── 10-training-loop.md — Forward pass, loss, backpropagation
├── 11-optimizers.md — SGD, Adam, AdamW, learning rate schedules
├── 12-data-pipeline.md — Datasets, loaders, preprocessing at scale
├── DATASET_GUIDE_EXTERNAL_SOURCES.md — Hugging Face, Kaggle, HTTP files, Common Crawl
├── DATASET_GUIDE_BUILD_FROM_SCRATCH.md — Collection, cleaning, dedup, splits, manifests
├── 13-distributed-training.md — Data parallel, model parallel, multi-node
PART 4: INFERENCE & DEPLOYMENT
├── 14-inference-engine.md — Prefill, decode, KV cache, continuous batching
├── 15-sampling.md — Temperature, top-k, top-p, speculative decoding
├── 16-quantization.md — INT8, GPTQ, AWQ, GGUF, QAT
├── 17-evaluation.md — Perplexity, MMLU, GSM8K, HumanEval, evals
PART 5: ADVANCED ARCHITECTURES
├── 18-advanced-attention.md — GQA, Gated DeltaNet, DeepSeek Sparse Attention, MLA
├── 19-multi-token-prediction.md — Predicting N tokens at once
├── 20-mixture-of-experts.md — Fine-grained MoE, shared experts, sparse dispatch
├── 21-thinking-engine.md — Chain-of-thought, token budgets, reasoning modes
├── 22-kernels.md — Custom CUDA, SIMD, FlashAttention in C++
PART 6: FINE-TUNING & ALIGNMENT
├── 23-fine-tuning.md — SFT, LoRA, QLoRA, DoRA, QDoRA
├── 24-grpo-dpo-rlhf.md — GRPO, DPO, RLAIF, reward models
├── 25-distillation.md — On-policy and offline distillation
PART 7: MULTIMODAL
├── 26-vision.md — ViT, CLIP, projectors, LLaVA-style fusion
├── 27-audio.md — Mel-spectrograms, frozen encoders, audio understanding
├── 28-video-documents.md — Temporal fusion, layout awareness, page rasterization
PART 8: PRODUCTION SYSTEMS
├── 29-safety.md — Prompt injection, jailbreak, PII, toxicity, red-teaming
├── 30-serving.md — HTTP server, OpenAI API, prefix caching, multi-tenant
├── 31-rag.md — Retrieval-Augmented Generation, embedding, ANN index
├── 32-tool-use-agents.md — Tool calling, sandboxed execution, multi-agent orchestration
├── 33-model-merging.md — SLERP, task vectors, weight averaging
PART 9: CONTINUOUS LEARNING
├── 34-self-learning.md — Online GRPO, replay buffer, self-critique
├── 35-forgetting.md — Catastrophic forgetting diagnostics, Manas integration
PART 10: THE COMPLETE PICTURE
├── 36-end-to-end.md — Full pipeline: idea → data → train → deploy → iterate
├── 37-v4-vision.md — Experimental roadmap and required ablations
RUNNABLE REFERENCE AND VALIDATION
├── code/minillm.py — Small real Transformer training + generation
├── tests/test_minillm.py — Unit tests for updates and checkpoints
├── tools/validate_markdown.py — Syntax, fence, and local-link validation
├── REFERENCES.md — Primary-source index by topic
└── VALIDATION_REPORT.md — Automated and runtime check summary
How to read the learning map: Move from top to bottom when learning from scratch. Parts 1–4 form the core implementation path; Parts 5–9 extend into research and production concepts; Part 10 combines the system and clearly separates a runnable mini-model from speculative v4 work. File indentation indicates grouping, not software dependencies.
Every chapter follows the same structure:
- Definition — One-sentence technical definition
- Beginner Explanation — Plain language, no jargon
- Why We Need It — The problem this solves
- Under the Hood — Mathematical derivation, step-by-step computation
- Python Code — Explicitly labelled as executable, educational, simulated, or experimental
- C++/CUDA Code — Where performance matters
- Walk-Through Example — Concrete numbers, traced through the entire computation
- Diagram — Visual representation followed by a written reading guide and limitations
- Common Beginner Questions — FAQ with detailed answers
- Key Vocabulary — Terms defined in context
- What's Next — Preview of the next chapter
# From the cloned repository root
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
# Validate the manuscript and run the real mini-model tests
make check
# Run the small CPU model and generate text
make demo
# Optional: run a longer educational training job
make trainPython blocks are syntax-checked, but some advanced blocks are educational sketches and are not independently production-ready. The supported end-to-end CPU example is code/minillm.py; chapter status notes explain the scope of every other block.
Use these after Chapter 12 or whenever you are ready to replace toy text with a documented corpus:
- Using External Datasets from Hugging Face and Other Sources — inspect licenses and schemas, pin revisions, stream large datasets, load local/remote files, use Kaggle and Common Crawl responsibly, normalize records, and create manifests.
- Building Your Own Dataset from Scratch — define a task and schema, collect authorized files, preserve provenance, clean text, remove exact duplicates, create deterministic leakage-safe splits, export JSONL, and document the release.
Both guides include explained diagrams, executable Python examples, reproducibility rules, privacy and licensing warnings, troubleshooting, and release checklists.
Start with → Chapter 1: What is Artificial Intelligence?
The chapters are designed to be read in order, but each is self-contained enough to jump into if you have prior knowledge.
Technical corrections, primary-source references, diagram improvements, and reproducibility fixes are welcome. Read CONTRIBUTING.md before opening a pull request. Use the issue forms for reproducible bugs and content corrections.
This repository is educational and is not hardened for untrusted production workloads. Report security vulnerabilities privately according to SECURITY.md.
Citation metadata is available in CITATION.cff. GitHub can generate BibTeX and other citation formats from the repository's Cite this repository control.
Copyright 2026 Darshan Vichhi and contributors.
Licensed under the Apache License 2.0.