Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions book-id/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
images/cover-image.png
43 changes: 43 additions & 0 deletions book-id/afterword.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Afterword: Back to Agent = LLM + Context + Tools {.unnumbered}

This book opened with a formula: **Agent = LLM + Context + Tools**. All ten chapters unfold within these three words.

**Chapter 1** establishes a three-layer understanding of the formula—the implementation layer, the intuitive layer, and the academic layer—and presents the orchestration spectrum from workflows to autonomous Agents. The chapters that follow unfold progressively along the sequence “Building—Evaluation and Evolution—Interaction and Collaboration.”

- **Building Agents (Chapters 2–5).** Context engineering determines what an Agent sees within a task, while memory and knowledge bases extend information across sessions. Tools define what it can do, and code generation provides the meta-capability to create new tools and systems.
- **Evaluation and Evolution (Chapters 6–8).** Evaluation turns performance into trustworthy signals, post-training writes high-dimensional capabilities into model parameters, and continuous evolution transforms production experience into controlled updates to knowledge, instructions, programs, or parameters.
- **Interaction and Collaboration (Chapters 9–10).** Multimodal and real-time interaction extends perception and action to speech, GUIs, and the physical world, while multi-agent collaboration further changes how context, tools, and responsibility are organized.

These three levels are not independent shelves. Chapter 8, in particular, depends on all the foundations established earlier: without trajectories and knowledge systems, experience has nowhere to be stored; without code capability, an Agent cannot modify tools and Harnesses; and without evaluation, the system cannot determine whether a modification is progress or regression. Chapter 8 is therefore the convergence point at which the book shifts from “how to build an Agent” to “how to make an Agent improve over the long term.”

## Two Clouds {.unnumbered}

In 1900, Lord Kelvin said two clouds still hung over the clear sky of physics—later, one became relativity and the other quantum mechanics. Today the sky over Agents is hardly clear either, and I too see two clouds.

**The first cloud is how Agents can interact with their environment in a streaming, real-time way.** Today, the vast majority of Agents still operate in a turn-by-turn "request-response" mode: you finish a sentence, it thinks through an entire paragraph, and then spits out the result all at once. But the real world doesn't stop and wait for it to finish thinking—speech gets interrupted, the scene keeps changing, emails keep arriving. A truly "living" Agent should be able to listen while thinking and speak while thinking, start planning while you're still mid-sentence, and notice on its own that "this email needs handling" even when no one has asked. There are two paths toward this real-time capability, often pursued in parallel. One is **architectural separation of fast and slow**—real-time responsiveness and intelligence are nearly orthogonal axes that a single model struggles to span, so a fast frontend model keeps the conversational rhythm while a slow backend model does the deep thinking. The other is **making inference itself faster**—when decode speed is high enough, the turn-by-turn wait becomes so short it nearly disappears, blurring the line between turn-by-turn and "real-time." This path is being rapidly advanced by chips and inference engines: Xiaomi MiMo has pushed a 1T-parameter model past 1000 token/s on a single 8-GPU node[^mimo], while dedicated solutions that hardcode the entire model into a chip (like the Taalas HC1) push an 8-billion-parameter model to about 17000 token/s with a response time under 100 milliseconds[^taalas]. When a model can spit out thousands of words per second, the experiential gap between "think, then speak" and "think while speaking" is simply erased.

**The second cloud is how Agents can, like humans, keep accumulating experience from the successes and failures of their interactions with the environment.** Today's models are more like a genius with a superb memory who can't learn anything new: during training they memorize human knowledge down to the last detail, but once on the job they barely grow—after each task, the pitfalls they've stepped in and the tricks they've figured out are mostly discarded along with the context. Whether this is a real problem depends on two opposing hypotheses.

One is the **"Small World Hypothesis"**: a sufficiently large model—say, with trillions of parameters—already contains almost all the important general knowledge of the physical world; learning once is enough. Those who hold this view (including researchers at OpenAI and Anthropic) would point out that programming is the one domain where AI is strongest today—not because code is somehow special to models, but because programming is humanity's most open field: vast amounts of open-source code sit there, ready to be learned from, while most industries have no public information or data at all. So what the frontier labs are really doing is going industry by industry, partnering with each to "distill" its professional capability into the same large model. According to this view, the bottleneck is neither the model's capacity nor its ability to learn, but whether there's enough data—feed the data in, train once, and the problem is solved.

But the **“Big World Hypothesis”** points to a layer that cannot be supplied by “training once”: knowledge specific to a particular user or company. Your company's coding standards, your team's taste in PPTs, and a particular client's distinctive temperament are absent from every training corpus and change constantly. To fit this “big world” composed of countless specific situations, a model must continue learning after deployment; it cannot arrive from the factory with everything configured. This is precisely the direction explored by memory in Chapter 3 and continuous evolution in Chapter 8: should experience be written into knowledge documents, instructions, or programs, or should selected experience be used to update model parameters? More broadly, both “AI for AI” and “AI for Science” are pushing Agents toward frontiers where no ready-made answers exist. There, an Agent can only learn autonomously from repeated experimental successes and failures rather than turning back to humans for every decision. The model's strongest capability will therefore ultimately be not memorization, but learning and adaptation.

Neither cloud will be blown away by any single model upgrade. To understand how they will eventually be dispelled, we first need to see one thing clearly: models and Agents have never been upstream and downstream of each other; they move forward together.

## The Co-Evolution of Models and Agents {.unnumbered}

Look back at the layers of fallback logic in those harnesses—multi-level context compression, retry logic that trips a circuit breaker only after thousands of failures, permission checks that pessimistically default to "unsafe"—and every stretch of seemingly ugly "spaghetti code" records a place where the model is still shaky. When the next generation of models internalizes these constraints, the corresponding code can be deleted; and the reason models can internalize them is precisely that Agents have already stumbled through those pits on the model's behalf in real business, condensing the lessons into signals for the next round of training. Users pose real challenges; the application layer uses harnesses to patch over what the model can't yet do well; those patches in turn become training signals for the model's next iteration. This is a self-reinforcing flywheel, and it is this flywheel that gradually pushes the two clouds away.

This flywheel also answers the question left hanging in Chapter 1: **will models eventually eat the Harness? Yes—layer by layer.** Every capability a model stably internalizes lets the corresponding Harness layer be deleted—Chapter 9's interaction model is one such case: behaviors like interruption and interjection that once had to be assembled with an external harness are now built directly into the model. But this "eating" will never be finished. First, training takes months—the model can wait, but the business cannot. Second, a model cannot internalize every constraint and preference of real business; there is always a newest boundary that needs external logic as a backstop. Third, every generation of models opens a new capability frontier, and the frontier is exactly where the model is least reliable. So the Harness will not disappear; it simply keeps migrating, together with the model, toward each new frontier. This is also how the Bitter Lesson reads in the Agent era: general methods will win in the end—but every stretch of road inside that "in the end" is paved by the Harness.

And the flywheel spins fastest in the hands of those who hold both ends. What Anthropic is doing with Claude Code is exactly this: letting its own model and its own harness feed each other and co-evolve. The model knows how the harness will call it; the harness knows where the model's boundaries lie; every change on either end feeds back to the other immediately. In one experiment, changing nothing but the harness—same model—lifted task accuracy from 52.8% to 66.5%. That shows how much leverage the harness has today, but it is also a reminder: the harness has that much leverage precisely because the model hasn't gotten there yet. And for the same reason, this flywheel itself is the deepest moat of this era: the tighter real business, feedback data, and model iteration mesh together, the harder it is for anyone to catch up from the outside.

What this means for you depends on which end of the flywheel you stand on. If you're building models, the moat is to get this flywheel spinning—let feedback from real-world scenarios flow back into training as quickly as possible. If you're building applications on top of models, the harness is your sharpest short-term technical lever, but be clear-eyed: each time the model internalizes a layer of constraints, it will—almost in passing—wipe out a batch of advantages built on the harness alone. The truly lasting moats at the application layer usually lie outside technology: exclusive data, solid distribution channels, user trust, network effects, and the complex scenarios that AI can't yet handle and that require humans and Agents working together. The prudent play is to use the harness to buy time, and use that time to build barriers beyond technology.

So don't fret over whether the framework in your hands will become obsolete. Models iterate every few months; specific APIs, products, and leaderboards will all turn over. But the three questions—what it sees, what it can do, and how to verify it's doing things right—will not become obsolete. They describe not the usage of a particular model, but the fundamental way an intelligent system interacts with the world. Master them, and whatever new capability the next generation of models brings, you'll know where it fits in the formula—and you'll see at a glance how far it still is from blowing those two clouds away.

Agent technology is still evolving at full speed; no single book can keep up with every change. But if what this book leaves you with is not the specific usage of some API but the judgment to stay clear-headed amid the waves of technology, then it has fulfilled its mission. All of this book's text, illustrations, and companion experiment code are open source. You're welcome to visit the repository, run the experiments with your own hands, and submit issues and PRs—between understanding and building lies a river that can only be crossed by hand. And the most fascinating thing about Agents is precisely this: they can create new capabilities by writing code, and even improve themselves. Having read this far, you already hold the principles of "creation." Now, go create something.

[^mimo]: Xiaomi MiMo-V2.5-Pro-UltraSpeed, through model-system co-design with FP4 quantization, DFlash parallel speculative decoding, and the TileRT inference system, pushes the generation speed of a 1T-parameter model past 1000 token/s on a single general-purpose 8-GPU node for the first time. See Xiaomi MiMo official technical blog, "Pushing 1T-Parameter Model Generation Speed to 1000 TPS," 2026. https://mimo.xiaomi.com/blog/mimo-tilert-1000tps

[^taalas]: The Taalas HC1 hardcodes the entire Llama 3.1 8B model onto a 6nm chip, achieving approximately 17000 token/s with a response time under 100 milliseconds; the trade-off is that the chip can only run the hardcoded model, and model updates require a new chip tape-out. See Karl Freund, "Taalas Launches Hardcore Chip With 'Insane' AI Inference Performance," Forbes, 2026. https://www.forbes.com/sites/karlfreund/2026/02/19/taalas-launches-hardcore-chip-with-insane-ai-inference-performance/
91 changes: 91 additions & 0 deletions book-id/build_pdf.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
#!/bin/bash
# Build the complete book as a single PDF (ElegantBook design, teal/cyan theme).
# Requirements: pandoc, xelatex, ElegantBook class, rsvg-convert (librsvg),
# fonts: Menlo / Arial Unicode MS (macOS) — Linux auto-falls back
# to DejaVu Sans/Mono and Noto CJK (see preamble.tex).
# Usage: cd book && bash build_pdf.sh
# Note: chapter/section numbers come from the document class; source headings
# carry no manual numbers (see git history for the de-numbering pass).

set -e

SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
cd "$SCRIPT_DIR"

# ── Runtime environment tweaks (harmless on macOS, needed on Linux/TeX Live) ──
# 1) This book is large enough to exhaust XeTeX's default main memory
# ("TeX capacity exceeded ... [main memory size=5000000]") during page
# output. main_memory is baked into the format at dump time, but
# extra_mem_top/bot extend an existing format at runtime — so we bump them
# here instead of rebuilding the xelatex format.
export extra_mem_top=8000000
export extra_mem_bot=8000000
# 2) The mac-only monospace font (Menlo) is probed with \IfFontExistsTF in
# preamble.tex. On systems without it, kpathsea otherwise spawns METAFONT to
# build a Menlo.tfm (slow, noisy, always fails) before the DejaVu Sans Mono
# fallback engages. Disabling on-the-fly TFM creation makes the probe return
# immediately with the correct "not found" result.
export MKTEXTFM=0

OUT="AI-Agents-in-Depth-Bojie-Li-v1.3.pdf"
CHAPTERS=(
introduction.md
chapter1.md
chapter2.md
chapter3.md
chapter4.md
chapter5.md
chapter6.md
chapter7.md
chapter8.md
chapter9.md
chapter10.md
afterword.md
)

# Verify all chapters exist
for ch in "${CHAPTERS[@]}"; do
if [ ! -f "$ch" ]; then
echo "Error: $ch not found" >&2
exit 1
fi
done

echo "Building PDF from ${#CHAPTERS[@]} files..."

pandoc "${CHAPTERS[@]}" \
-o "$OUT" \
--from markdown+lists_without_preceding_blankline \
--pdf-engine=xelatex \
--lua-filter=crossref.lua \
--lua-filter=experiment_box.lua \
--toc \
--toc-depth=3 \
--number-sections \
-V documentclass=elegantbook \
-V classoption=lang=en \
-V classoption=cyan \
-V classoption=device=normal \
-V author="Bojie Li" \
--metadata title-meta="AI Agents in Depth: Design Principles and Engineering Practice" \
--metadata author-meta="Bojie Li (English translation: Devaraj)" \
-H preamble.tex \
--include-before-body=cover.tex \
--highlight-style=kate \
--columns=80 \
2>&1

if [ -f "$OUT" ]; then
SIZE=$(du -h "$OUT" | cut -f1)
PAGES=$(python3 -c "
import subprocess, re
r = subprocess.run(['pdfinfo', '$OUT'], capture_output=True, text=True)
m = re.search(r'Pages:\s+(\d+)', r.stdout)
print(m.group(1) if m else '?')
" 2>/dev/null || echo "?")
echo ""
echo "Done: $OUT ($SIZE, $PAGES pages)"
else
echo "Error: PDF generation failed" >&2
exit 1
fi
Loading