The LLMs-Journey repository covers topics including:
- Large Language Models (LLMs)
- Agentic Systems and Workflows
- Fine-Tuning (e.g., LoRA, QLoRa)
- Retrieval-Augmented Generation (RAG)
- Vision-Language Models (VLMs)
- Complementary Resources and Research
Fine-Tuning:
Agents:
- Fine-Tuning LLMs for Function-Calling with MLX-LM
- A Job Postings Tool: A Guide to MLX-LM Server and Tool Use with the OpenAI Client
- Running Local Hugging Face Models with MLX-LM and the Agno Agentic Framework
VLMs:
- Qwen2.5-VL with MLX-VLM
- Image Segmentation with PaliGemma 2 Mix and MLX
- Image Segmentation with PaliGemma 2 mix, Transformers, Docker, FastAPI, and GitHub Actions
- Nanonets OCR: A Small Gem for Handwritten Notes
On-Device AI:
Misc:
Books:
- AI Engineering by Chip Huyen
- Hands-On Large Language Models by Jay Alammar, Maarten Grootendorst, GitHub
- THE HUNDRED-PAGE LANGUAGE MODELS BOOK by Andriy Burkov, GitHub
- Foundations of Large Language Models
- How to Scale Your Model
- The Ultra-Scale Playbook: Training LLMs on GPU Clusters
Agents:
- Hugging Face Agents Course
- ai-agents-for-beginners (Microsoft)
- Agents (Google's whitepaper)
- Agency Is Frame-Dependent
- Agent Lightning: Train ANY AI Agents with Reinforcement Learning
Promptning:
Fine-Tuning:
- 🤗 PEFT: Parameter-Efficient Fine-Tuning of Billion-Scale Models on Low-Resource Hardware
- LoRA: Low-Rank Adaptation of Large Language Models
- QLORA: Efficient Finetuning of Quantized LLMs
RAG related:
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Chain-of-Retrieval Augmented Generation
- Introducing Contextual Retrieval
Evaluation:
- A Survey on LLM-as-a-Judge
- ARC Prize 2024: Technical Report
- FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
- MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Datasets:
Models:
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- RoFormer: Enhanced Transformer with Rotary Position Embedding and Round and Round We Go! What makes Rotary Positional Encodings useful?
- Qwen2.5-Math and paper
- Kimi K2: Open Agentic Intelligence
Vision:
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- Qwen2.5-VL and paper
- Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- Magma: A Foundation Model for Multimodal AI Agents
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
- PaliGemma 2: A Family of Versatile VLMs for Transfer
- Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks
- Kimi-VL Technical Report
- InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
- FastVLM: Efficient Vision Encoding for Vision Language Models
- DINOv3
- Qwen3-VL Technical Report
- SAM 3: Segment Anything with Concepts
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
- LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
- VL-JEPA: Joint Embedding Predictive Architecture for Vision-language
- DeepSeek-OCR: Contexts Optical Compression
- olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models
- olmOCR 2: Unit Test Rewards for Document OCR
- dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model
- Can Large Vision Language Models Read Maps Like a Human?
Chain-of-Thought
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
Visualisation-of-Thought:
Test-Time Scaling
Test-Time Compute
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
AlphaGeometry:
Apple:
- Machine Learning Research at Apple
- Distillation Scaling Laws
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
- Reversal Blessing: Thinking Backward May Outpace Thinking Forward in Multi-choice Questions
- FastVLM: Efficient Vision Encoding for Vision Language Models
Misc:
- Scaling Pre-training to One Hundred Billion Data for Vision Language Models
- Competitive Programming with Large Reasoning Models
- MoBA: Mixture of Block Attention for Long-Context LLMs
- Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model, repo
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
- Deepseek Papers
- InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU
- Leveraging the true depth of LLMs
- NOLIMA: Long-Context Evaluation Beyond Literal Matching
- Memory Layers at Scale
- Towards an AI co-scientist
- On the consistent reasoning paradox of intelligence and optimal trust in AI: The power of 'I don't know'
- LIMO: Less is More for Reasoning
- How new data permeates LLM knowledge and how to dilute it
- Attention Is All You Need
- mHC: Manifold-Constrained Hyper-Connections
- GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization, GitHub
MLX:
LangChain:
Vector Databases:
HuggingFace:
Leonie Notebooks:
- Fine-tuning Gemma 2 JPN for Yomigana with LoRA
- Advanced RAG with Gemma, Weaviate, and LlamaIndex
- RAG with Gemma on HF 🤗 and Weaviate in DSPy
GitHub Repos:
- rag-cookbooks
- Hands-On-Large-Language-Models
- Multimodal-RAG-Implementation
- Data_Science_Learning_Material
- RAG-To-Know
- Docling
Blogs/Posts:
Frameworks: