research.md

Research Resources

This bibliography connects the ideas in the experiment guide to their reusable implementations. Experiment-specific caveats remain beside the runnable code; this page provides the broader papers, datasets, and technical references.

Core References

See it in the repo: matrix operations, tensor runtime, architecture, and the foundation experiments.

• Deep Learning - Feedforward networks, backpropagation, losses, optimization, and numerical computation.
• BLAS - Vector, matrix-vector, and matrix-matrix operation interfaces.
• LAPACK - Dense linear algebra routines built around optimized BLAS usage.
• Zig Language Reference 0.16.0 - Language reference for the Zig version used by the project.
• Zig Standard Library - Allocators, arrays, math functions, file I/O, and testing APIs.

Neural Network Foundations

See it in the repo: matrix, layers, networks, and the foundation experiments.

• Learning representations by back-propagating errors
  • Backpropagation.
• Understanding the difficulty of training deep feedforward neural networks
  • Xavier-style initialization and gradient flow.
• Delving Deep into Rectifiers - He, Zhang, Ren, and Sun's rectifier initialization paper.
• Gradient-Based Learning Applied to Document Recognition
  • Neural-network training for document recognition and MNIST-style data.

Optimization

See it in the repo: training and optimizers, reusable modules, and Optimizer Lab.

• Adam: A Method for Stochastic Optimization
  • Adaptive first- and second-moment optimization.
• Decoupled Weight Decay Regularization
  • AdamW and weight decay separated from the gradient update.
• On the importance of initialization and momentum in deep learning
  • Momentum behavior and optimization conditioning.

Spectral Learning

See it in the repo: spectral transforms and features, network fundamentals, and Spectral Learning.

• On the Spectral Bias of Neural Networks
  • Frequency-dependent learning speed and the tendency of deep networks to learn low-frequency structure first.
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
  • Coordinate feature mappings that expose tunable harmonic structure to an MLP.

Activations And Gating

See it in the repo: activation functions, gated layers, Gated Network, and the implementation notes.

• Gaussian Error Linear Units - Hendrycks and Gimpel's GELU paper.
• Searching for Activation Functions - Ramachandran, Zoph, and Le's Swish paper.
• Language Modeling with Gated Convolutional Networks
  • GLU.
• GLU Variants Improve Transformer - SwiGLU and other gated feed-forward variants.

Representations, Vision, And Audio

See it in the repo: spatial layers, audio features, reusable modules, and the vision and audio experiments.

• Gradient-Based Learning Applied to Document Recognition
  • Convolutional networks and handwritten-digit recognition.
• Extracting and Composing Robust Features with Denoising Autoencoders
  • Corruption-based representation learning and reconstruction.
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
  • Dataset construction and keyword-recognition evaluation.
• TensorFlow simple audio recognition tutorial
  • The Mini Speech Commands teaching subset and log-mel workflow used as a practical reference.

Transformers And Tiny GPT

See it in the repo: Transformer components, tensor primitives, TinyGPT, and the language and sequence experiments.

• Attention Is All You Need - Vaswani et al.'s Transformer paper.
• Improving Language Understanding by Generative Pre-Training
  • GPT-1.
• Language Models are Unsupervised Multitask Learners
  • GPT-2.
• minGPT - Educational GPT implementation.
• nanoGPT - Compact GPT training implementation.
• TinyStories paper - Small language models trained on simple stories.
• TinyStories dataset
  • Dataset card and files.

NLP, Structured Prediction, And Retrieval

See it in the repo: text processing, embeddings, structured prediction, decoding, retrieval, and the language and sequence experiments.

• Neural Machine Translation by Jointly Learning to Align and Translate
  • Encoder-decoder attention and learned alignments.
• Efficient Estimation of Word Representations in Vector Space
  • Skip-gram Word2Vec.
• Distributed Representations of Words and Phrases and their Compositionality
  • Negative sampling and subsampling for word representations.
• Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
  • Linear-chain conditional random fields.
• Neural Machine Translation of Rare Words with Subword Units
  • Byte-pair encoding for subword tokenization.
• The Curious Case of Neural Text Degeneration
  • Nucleus top-p sampling.
• Learning Transferable Visual Models From Natural Language Supervision
  • Symmetric contrastive learning across paired encoders.

Sequence Models And Reinforcement Learning

See it in the repo: recurrent models, reinforcement-learning utilities, GRU Sequence, and DQN.

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
  • The gated recurrent unit and encoder-decoder recurrence.
• Playing Atari with Deep Reinforcement Learning
  • Experience replay, target networks, and deep Q-learning.
• Human-level control through deep reinforcement learning
  • The later DQN evaluation and algorithm presentation.

Quantization

See it in the repo: quantization primitives and the TurboQuant experiment.

• TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
  • Random rotation, scalar quantization, inner-product error, KV-cache, and nearest-neighbor quantization.
• Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
  • Integer-only neural-network inference.
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
  • Transformer inference quantization and outlier-aware mixed precision.
• GPTQ - Post-training weight quantization for GPT-style models.
• SmoothQuant - Activation-outlier smoothing for efficient LLM inference.
• A Survey of Quantization Methods for Efficient Neural Network Inference
  • Quantization terminology and method taxonomy.

GPU And Backend Work

See it in the repo: backend abstraction, Metal, CUDA, ROCm, the runtime experiments, and the GPU verification guide.

• Apple Metal - Metal overview, platform support, and tooling.
• Performing calculations on a GPU
  • Metal command queues, buffers, compute pipelines, and dispatch.
• Metal-cpp - Official C++ interface for Metal.
• CUDA C++ Programming Guide
  • CUDA execution model, memory model, and programming interface.
• CUDA C++ Best Practices Guide
  • CUDA optimization, profiling, memory hierarchy, and numerical accuracy.
• cuBLAS - NVIDIA BLAS library documentation.
• ROCm HIP documentation
  • AMD HIP programming model and runtime documentation.
• HIPRTC
  • Runtime compilation for HIP kernels.
• rocBLAS - AMD BLAS library documentation.

Serving And Tooling

See it in the repo: inference service, XOR serving, and the TinyGPT OpenAI-compatible server.

• TensorFlow Serving: Flexible, High-Performance ML Serving
  • Model serving architecture, versioning, model lookup, and lifecycle.
• TensorFlow Serving docs - Serving overview and API references.
• NVIDIA Triton Inference Server docs
  • Model repositories, batching, backends, and inference serving.
• Go net/http - Official HTTP package docs for servers and clients.
• Go encoding/json - Official JSON package docs.

Datasets

See them in the repo: MNIST, Speech Commands, and the TinyGPT corpus notes.

• MNIST database - Original MNIST dataset home.
• Speech Commands dataset - Dataset design, collection, and evaluation conventions.
• TinyStories dataset
  • Dataset card and files.
• Tiny Shakespeare corpus
  • Small character-level language-modeling corpus.