Agents and AI system optimizationAGENTA study handbook
This handbook treats an agent as a systems problem. The model sets the ceiling, the harness sets real performance, and the serving system and GPU set cost and speed. Chapters 1 to 8 cover using and designing agents: the loop, tools, context, memory, multi-agent systems, reliability, evaluation and cost. Chapters 9 and 10 cover how LLM inference and GPUs work. Each chapter opens with key points, then the material, then concrete steps for putting it to work, and ends with a self-check.
SCROLL
01CHAPTER 1
The big picture: AI systems and agentic systems全景：AI 系统与智能体系统
Key points
AI systems is the study of how to train and run models at scale so they are fast, cheap, reliable and sustainable; in this handbook it means large language model (LLM) systems.
An agent uses a model to reason, call tools and act step by step until a task is done. Think of it as a new kind of computer: the LLM is the CPU, context is RAM, tools are peripherals, the harness is the operating system.
A running agent is a long chain of inference and tool calls, and each property of that workload names a systems problem to solve.
The thing to optimize is not tokens per second but getting the task done within limits on latency, cost, resources, reliability and security.
Two words come up in every chapter. A token is the smallest unit of text a model reads or writes, roughly a word or part of one; price and speed are both counted in tokens. The harness is the program wrapped around the model: it packages your request and files for the model, runs the tool calls the model asks for, checks permissions and feeds the results back. The same model can perform very differently in different harnesses.
Five ways AI systems differ from traditional systems
Dimension
What differs
Compute
The CPU and GPU split the work. The CPU tokenizes text, launches the small programs that run on the GPU (kernels), moves data and runs tool calls; the GPU does large matrix multiplications. Reading the whole prompt is called prefill and is limited by compute; producing tokens one at a time afterwards is called decode and is limited by how fast memory can be read (bandwidth). How long a request takes depends on how long the output is, which is unknown in advance, so scheduling is hard
Non-determinism
The same input can give different outputs. Adding floating-point numbers in a different order gives slightly different results, and a shift in the seventh decimal can flip the next token. The variation comes from the runtime (the order in which GPU thread groups, called warps, are scheduled; how many requests share a batch), software (whether multiply and add are fused into one FMA step; which algorithm autotuning picked) and hardware (ordinary cores or tensor cores; low-precision formats such as FP8). This approximation is acceptable
Memory, storage, communication
Model weights run from GBs to TBs. The KV cache is what the model keeps for every token it has already read so it need not recompute it, and it takes tens to hundreds of KB per token; a prefix cache lets requests that start the same way share it. Moving data, not computing, is often the bottleneck. When one machine is not enough, the model is split across GPUs (by layer, by matrix, by expert, by context) over fast interconnects such as NCCL, NVLink, InfiniBand and RDMA
Fast-moving models and workloads
Attention, the mechanism a model uses to find what matters in its context, moved from MHA to GQA, MLA, NSA and linear attention, mostly to save memory and compute. Workloads moved from multi-turn chat, long reasoning and retrieval-augmented generation (RAG) to agents, which need long context, prefix caching, and routing and load balancing across models and machines
Reliability and power
GPUs fail at 9% a year, so a 16K-GPU cluster loses one about every 3 hours. A GPU draws 700 to 1000 W, and server power rose from 800 W to 6,000 W
Server power rose about eightfold with GPUs
Power per server, W
CPU server800 W
GPU server6,000 W
Original lecture notes
Chapters 9 and 10 go into these details. For now, remember one thing: longer outputs are slower and costlier, and two runs are not guaranteed to agree.
What an agent is
An AI agent is an autonomous software system that uses an AI model to reason about a goal, use external tools (search, reading and writing files, running code, calling APIs) and carry out multi-step actions until the task is finished. Its work breaks into five steps:
01Plan: break the task into sub-steps
→
02Act: call external tools
→
03Observe: read what the tools returned
→
04Adapt: revise the plan on new information
→
05Deliver: return the final result
Agents exist because LLMs are uneven. They beat most people at recalling facts, language, pattern matching, writing code and connecting fields, but they forget past conversations, cannot reliably understand long context, state false things with confidence (hallucination), don’t know what they don’t know and are easily steered by hostile input. Tools give the model hands; memory gives it recall.
An agent can be read as a new kind of computer: the LLM is the CPU, the context window (all the text the model can see at once) is RAM, tools are peripherals, the harness is the operating system and the agent is the application. How you program it and how it fails are both entirely different from a traditional computer.
Why agents are a systems problem
Ordinary LLM serving is one request, one inference, one response. An agent is a chain: infer, call a tool, infer, call a tool, retrieve, infer. One task passes through GPU, CPU, storage, network and outside services, which brings six problems: each step depends on the last, step times vary widely, the agent must wait on tools, state keeps growing, it needs concurrency and guessing ahead, and anything along the chain can fail.
Property of the agent workload
Systems problem it names
Long trajectories (tens or hundreds of steps per task)
State and context management
Huge context
Memory hierarchy and retrieval
Repeated prefixes (every step carries the same system prompt and history)
Prefix caching
Branching reasoning
Speculative execution
Tool dependencies
Dependency-aware scheduling
Different models
Routing
Variable tool latency
Asynchronous execution
External side effects (sending email, changing a database)
Security and isolation
Multiple agents
Orchestration and communication
Long-running tasks
Checkpointing and fault tolerance
The stack, top to bottom: application and user; agent and harness (planning, context, memory, tools, subagents), beside the external environment and tools (web, files, APIs, humans); the model; the serving system (batching requests together, scheduling, routing, caching); runtime and kernels (CUDA, Triton, multi-GPU communication); hardware (GPU, CPU, memory, network, storage).
What to optimize
Throughput, tokens generated per second, is the wrong objective for an agent. The objective is to get tasks done within limits on latency, cost, resources, reliability and security.
What to optimize
Task success and accuracy
Time from request to result
Tokens per task
Dollars per successful task
Energy per successful task
Useful, but insufficient
TTFT (time to first token) and TPOT (time per output token)
Tool-call latency
GPU utilization
More tokens per second can still mean a slower, costlier agent, for example one that takes ten extra steps. That makes observability a first-class problem: a task must unfold into nested records of sessions, turns, agent steps, model requests, tool calls and subagents. You cannot optimize what you cannot see.
Questions that run through this handbook
Batching and latencyWhy does batching requests raise throughput but slow an agent down? When is a larger model worth it?
Where state livesWhere should the KV cache and shared prefixes live? How should requests from one agent be queued?
Overlap and delegationCan the model do useful work while a tool runs? When should an agent hand work to a subagent?
Routing and recoveryHow do we spread requests across different GPUs and models? How does a 30-minute task resume after a failure?
The objectiveHow do we optimize dollars per successful task rather than tokens per second?
This handbook leaves out model training (inference now dominates compute demand), model architecture design, multi-machine communication, power, and new hardware design.
Putting it to work
01Define what “done” means before you start: all tests pass, every report field is filled, figures match the source table. Without that test you cannot measure success rate or cost per successful task.
02Account per task, not per call: for each task record total tokens, spend, time and whether it succeeded. When you compare two agents or two prompts, compare these numbers, not which one replied faster.
03When a task is slow or expensive, open the agent’s run log or trace, unfold it into individual tool calls, and find the slowest, costliest or most repeated step before changing anything.
04Use the six levels below to decide how much to hand over. Start at level 3, handing off a well-scoped task and reviewing the result; move to level 4 once checking is automated with tests or validation scripts.
05Not everything needs an agent. Work with fixed rules that finishes in one step is cheaper and steadier as a script or a simple bot; chapter 8 covers how to tell.
Also: six levels of AI coding, and agents vs assistants vs bots
Level
Meaning
0
No AI, hand-written code
1
Assist: AI autocompletion
2
Supervised: AI writes functions, you review
3
Scoped: hand off a task, review the code
4
Goal-driven: AI writes all, tests verify
5
Parallel agents: subagents split the work
Agent
Assistant
Bot
Purpose
Performs tasks autonomously and proactively
Assists users with tasks
Automates simple tasks or conversations
Capabilities
Complex multi-step actions; learns and decides independently
Responds to prompts and can recommend; the user decides
Follows predefined rules; limited learning
Interaction
Proactive, goal-oriented
Reactive, responds to requests
Reactive, responds to triggers
Self-check
When you evaluate an agent, which number matters most?
Throughput is the wrong objective; TTFT and GPU utilization are useful but insufficient. The goal is getting tasks done within limits, measured as cost and time per successful task.
Self-check
Which systems problem does each workload property name?
Every step starts with the same text
Tools take unpredictable time
It changes outside data
A task runs for half an hour
02CHAPTER 2
How an agent works: the loop, memory and extensionsAgent 如何工作：循环、记忆与扩展
Key points
One request expands into many model calls and tool runs: the model proposes an action, the harness checks permission and runs it, the result goes back into context, and the loop goes round again.
Context is rented working memory that vanishes when the session ends; the project instruction file and auto memory are files loaded every session, and stale memory is worse than none.
Each extension does one job: a skill says how, MCP says what the agent can reach, a hook says what happens automatically at a given moment, a plugin packages it all for sharing.
A coding agent is, at heart, a loop:
01Gather context: your prompt, relevant files, the last tool result
→
02The model proposes an action, e.g. “run pytest”
→
03The harness checks permission and runs it
→
04The result goes back into context; repeat
Facts about the loop
Context holds over 10K tokens before you type. First come system instructions, the project environment, memory (the project instruction file), descriptions of the available tools (read files, edit, run commands, find files by name, search contents, browse the web, start a subagent), and the names and one-line descriptions of skills and MCP servers; your words come last. Skills and MCP start as names only; their full text loads when the agent reaches for them.
It does not send your whole codebase. The code stays on disk and the model pulls slices on demand: a search for “authentication” brings back only the matching lines.
The model proposes, the harness executes. The model emits a tool request; the harness parses it, checks permissions, runs it on your machine and puts the result (the tool result) back into the model’s context. Every request builds on the previous inference and the previous tool result.
Permissions sit between intent and execution. A tool request is only a proposal; you or another model must approve it (table below).
Context grows, then gets compacted. A typical window today is 1M tokens; 256K was common earlier in the year. Reading the cached prefix accounts for most of the cost. When the window fills, the agent compacts: it summarizes the old history and puts the project instruction file and memory back in.
Permission mode
Meaning
Manual
You confirm every step
Accept edits
File edits pass automatically; other actions still ask
Bypass permissions
Everything runs directly
Auto
A small model reviews each request
Each request ends one of three ways: allowed (the command runs), ask (you confirm first) or denied (nothing happens).
Memory: context is rented, memory is owned
An agent remembers nothing between runs; every session starts from zero. What we call memory is just files loaded at session start. Context is working memory and disappears with the session; persistent memory lives outside it and is reloaded next time. Memory solves four problems: giving the same instruction again and again, redoing the same exploration, losing decisions and their reasons at compaction, and different sessions choosing differently. Memory turns repeated reminders into convention.
Project instruction file
Auto memory
Who writes it
You
The agent
Contains
Instructions and rules
Lessons and patterns
Scope
Project, personal or organization
One per codebase, shared across its working copies
Loaded
Every session
Every session (only the start, e.g. the first 200 lines or 25KB)
Use for
Standards, workflows, architecture
Preferences, corrections, project background
The project instruction file is a Markdown file the agent reads at every start; the exact name and location depend on your agent’s documentation. It usually comes in four layers, loaded broadest first, with the most specific read last and winning: organization policy (cannot be switched off by individuals), your personal global settings, the project file (shared with the team through the codebase) and a local personal file (yours only, kept out of version control).
Belongs in the instruction file
Commands: how to run tests, the build and the linter
Hard constraints: never commit to main, never edit generated files
Facts the agent cannot cheaply find out: how folders are split, who owns what, known pitfalls
Does not belong
What the model already knows about the language or framework
Style a linter or formatter already enforces
Long prose, changelogs, anything secret
Keep each file under 200 lines, use Markdown headings and bullets, and make the rules concrete and consistent. When it grows, split it into one rule file per topic; if your agent lets you tag a rule with the paths it applies to, it loads only when the agent touches matching files.
Auto memory is written by the agent, and each entry usually carries a type: user (your role and preferences), feedback (approaches you corrected or confirmed), project (work in progress, decisions the code does not show) and reference (where outside material lives). An index file lists every entry and only its start loads each time; topic files are read only when relevant. Most agents offer commands to browse, edit or switch off memory, to see which instruction files are loaded, to draft an instruction file, and to check the file for content that does not belong.
Memory has a price: every resident token is rent paid at the start of every session, and it competes with the task for the same window. Stale memory is worse than none, because it actively misleads. Review it, prune it and keep it under version control, like code.
Tools and the four extension mechanisms
Core tool
Role
Read
Open files and inspect state; the most used
Search
Find by content (grep) or file name (glob) without reading everything
Fetch
Pull a web page or document; brings untrusted text into context
Edit / Write
Change files; the first tool with consequences
Shell
Build, test, git; without a sandbox it hands over the whole machine
Delegate
Start a subagent and get back a summary
The model decides which tool to use. To let the agent do more, there are four extension mechanisms:
Mechanism
Problem it solves
Where it attaches in the loop
Skill
How the agent should do a kind of task (knowledge and workflow)
Model reasoning
MCP
Which outside data and actions it can reach
Tool selection
Hook
What happens automatically at a given moment
Any stage; it can inspect, block or react
Plugin
How extensions are packaged and shared
Bundles the other three into one installable package
Take a release workflow (prepare, verify, publish to GitHub, notify the team):
01Skill: follow the release steps
→
02MCP: create the GitHub release
→
03Hook: block the release if tests fail
→
04Plugin: share the whole setup with the team
A skill is a folder: one instruction file (its name and description decide when the agent thinks of using it; the body is the workflow), plus reference material and scripts. Only the description loads at session start; the body loads when the skill runs. The model can invoke it on its own when the description matches the task, or you can call it by name with a slash command. Workflows with side effects should usually require an explicit call; clear, non-overlapping descriptions reduce wrong picks. At bottom, a skill is a way of managing context.
MCP (Model Context Protocol) connects outside services through one standard protocol: GitHub, databases, monitoring, Figma, Slack, browsers and more. A call takes six steps: the agent decides it needs an action; the MCP client calls a tool; the MCP server handles the connection; the outside service does the work; a structured result comes back; the agent evaluates it. In use it is almost the same as a local tool, only running in another process.
A hook is an automatic action attached to an event; it fires on time and does not depend on the model remembering. Model reasoning itself is not an event. A hook can run a command, send an HTTP request, call an MCP tool, insert a prompt or start a subagent.
01Session start: load context, log
→
02You submit a prompt: validate or add to it
→
03Before a tool call: inspect or block
→
04After a tool call: format, test, log
→
05Before stopping: confirm tests pass
A plugin bundles skills, hooks, subagent definitions, MCP settings and language-server settings into one package, which gives one-command install, pinned versions, names that do not collide, team reuse and distribution through a marketplace.
An agent’s configuration folder in a project usually has several parts: the instruction file and rule files are memory; a settings file holds permissions, hooks and environment variables; skills and custom commands are workflows; subagents and hooks are delegation and guardrails. Your personal folder has the same structure and applies to all your projects; project settings win over personal ones.
Need
Use
Follow the company API conventions
Skill
Read issues from Jira
MCP
Run the formatter after every edit
Hook
Guide a multi-step deployment
Skill
Send a notification when done
Hook plus MCP
Block writes to production
Hook or policy
Share the whole setup across codebases
Plugin
Putting it to work
01Write an instruction file for the project you use most: first the commands for tests, build and checks, then two or three hard constraints, then the folder layout and known pitfalls. Delete whatever the model already knows and keep it under 200 lines.
02Once you have corrected the agent on the same thing twice, put it in the instruction file or have the agent save it to memory. It is the cheapest improvement there is.
03Pick the extension by need: a workflow you repeat becomes a skill; reading or writing an outside system means MCP; anything that must happen every time and cannot rely on the model remembering (formatting, blocking dangerous commands) is a hook; package it as a plugin when the whole team needs it.
04Start permissions at manual or accept edits; use bypass only in a sandbox, a container or an environment you can throw away.
05Go through auto memory and the instruction file now and then, delete stale entries, keep changes under version control and review them as you would code.
Further: MCP in detail
Before MCP, connecting N AI clients to M services took N×M integrations. MCP is an open, stateless protocol: the host (for example, your agent) runs one MCP client per server, and an MCP server exposes a system’s capabilities as tools.
MCP primitive
What it is
Example
tools
Actions the model can call
create_issue(…)
resources
Data to read, such as an issue or a table schema
github://issues/123
prompts
Prompt templates the server provides; you decide when to use them
Shown in the agent as a slash command, e.g. “triage issue”
The three work together: to triage issue #123, the prompt supplies the organization’s triage procedure, the resource supplies the issue, and a tool fetches current diagnostics.
Security: least privilege, with tokens scoped to the repositories, tables or channels you need; separate reads from writes and require approval for writes; never treat tool output as instructions; review third-party servers before installing them, since they can carry prompt injection (instructions hidden in content to steer the model). Before connecting, ask two questions: what can this server read, and what can it change?
An MCP gateway is a control layer between AI clients and many MCP servers. It aggregates, routes, enforces policy, acts on behalf of the user’s identity, observes and scales. It pays off when there are many users, agents, servers or compliance rules, but the gateway is itself a trust boundary, with risks of over-broad credentials, losing the end user’s identity, tool-name collisions, secrets leaking into logs and a single point of failure. The 2026-07-28 specification is the largest revision since launch: the core became stateless, so any request can land on any instance; a new Mcp-Method header lets a gateway route without reading the request body, and tool lists can be cached for a set lifetime; long-running tasks (Tasks) and server-rendered interfaces (MCP Apps) ship as extensions.
Self-check
Design exercise: which mechanism fits each need?
Follow the company API conventions
Read issues from Jira
Run the formatter after every edit
Guide a multi-step deployment
Block writes to production
Share the whole setup across codebases
A skill holds knowledge and workflow, MCP gives outside access, a hook runs automatically on an event, a plugin packages and shares. Blocking writes to production can also be done with a permission policy.
Self-check
Which of these belongs in the project instruction file?
The file holds commands, hard constraints and facts the agent cannot cheaply find out; what the model knows, what tools already enforce, long prose and changelogs do not belong.
03CHAPTER 3
Multi-agent use and Programming 3.0多智能体用法与 Programming 3.0
Key points
Changing instructions alone can change how an agent works: Superpowers uses skills plus a session-start hook to impose a development process, and Caveman uses skills, hooks, MCP and a local proxy to cut tokens in both directions.
There are three ways to scale: parallel agents save waiting time, subagents buy focus, long runs widen what one request can cover; a team of agents is worth it only when the work needs discussion.
The core of Programming 3.0 is uncertainty: don’t expect the agent to behave the same every time, make the system around it reliable, and match autonomy to risk.
Field data show the tasks handed to agents grew sharply in six months, but the longer the task, the lower the share that succeeds without a human stepping in.
Case one: Superpowers
Superpowers is a plugin for software development. By default an agent starts writing code as soon as it gets a request and fixes mistakes by trial and error. With the plugin, the work follows a fixed path:
01Human types the request
→
02Clarify questions
→
03Approve a design
→
04Write a detailed plan
→
05Test-driven development: write a failing test first, then code until it passes
→
06Review and verify
→
07Finish or merge the branch
It is made of nine skills:
Skill
Role
using-superpowers
Rules for when to discover and invoke which skill
brainstorming
Turn a request into an approved design
writing-plans
Break the design into concrete tasks
using-git-worktrees
Set up workspaces that don’t interfere
test-driven-development
Red (test fails), green (test passes), refactor
systematic-debugging
Find the root cause before proposing a fix
requesting-code-review
Review the work against the plan
verification-before-completion
Show evidence before reporting “done”
finishing-a-development-branch
Merge, open a PR, keep or clean up the branch
A skill file has six parts: name, description, instructions, checkpoints, references and scripts. A process skill can also state entry conditions, required actions, forbidden actions, gates that need human approval, verification requirements and the next skill to use.
Installing skills does not guarantee the agent will look at them. Superpowers adds a hook that runs at session start: whenever a session starts, is cleared or is compacted, a script reads the full text of a “bootstrap” skill and injects it straight into context, reminding the agent to follow the process.
Design question
Superpowers’ answer
How is complex behavior represented?
Small, composable skills
How does the agent remember to use them?
A session hook injects a bootstrap skill
How does context stay manageable?
Task skills load only when relevant
How do stages connect?
Each skill names its gates and the next skill
How do users install it?
Everything ships as a plugin
Are commands or MCP required?
No; add them when needed
Case two: Caveman
Caveman compresses an agent’s tokens in both directions. There are three reasons: logs, diffs (side-by-side code changes) and tool output crowd the context; every turn costs money and waiting; and trimming only when the context fills cuts out key details. For example, 786 lines of test output become one line: “3 tests failed: ETIMEDOUT at client.ts:88”.
65%Reported cut in output tokens; the skill itself adds input tokens, so whole-session savings are smaller
2Directions: the skill changes how the model writes, the proxy changes what it reads
5MCP tools: compress, fetch the original, stats, and JSON-to-TOON conversion both ways
It uses all five extension mechanisms plus a local proxy (a small program that sits between the agent and the model service and can rewrite every request):
A skill sets a terse style: keep technical terms, commands, paths, code, errors, numbers, units and negations; drop filler, repetition, pleasantries and hedging.
Commands let you choose a level, such as /caveman ultra or /caveman off.
A session-start hook decides and saves the session’s level and restores the rules after compaction; a hook on prompt submission catches mid-session switches.
MCP provides compression and retrieval of the original.
A plugin packages it all.
The proxy compresses by content type: logs keep errors, stack traces and the first and last lines; code keeps imports, function signatures and types; diffs keep file names, hunk headers and changed lines; JSON keeps keys and structure.
A hook can intercept a request but cannot quietly rewrite what is sent to the model, hence the proxy. The proxy intercepts every request and changes the context; MCP runs only when called and cannot change context. If you write your own harness, you can do it directly.
Three ways to scale
Parallel agents
Subagents
Long runs
How
Independent tasks, one worktree each
Give a well-bounded task to a fresh context and get a summary back
One large task runs over many steps
You gain
Less waiting, not better answers
Time and focus: the main conversation never sees the messy middle
Scope: more done per request, less intervention
Parallel by hand. A git worktree is a second working folder of the same codebase: its own files and branch, a shared commit history, no touching each other’s files, no stashing and no second clone. Many agents can create a worktree per session with one command, say one for a payments feature and one for a login bug. With three sessions you can still hold the plan in your head. With twenty you need a copy of yourself: the plan lives only in your head, the sessions cannot see each other, you merge the branches yourself, and nothing is recorded to rerun.
Subagents are specialized assistants with their own context, tools and permissions, used for five things: keeping long output out of the main context, restricting tools and models, reuse across projects, parallel work and controlling cost (for example with a small model). A custom subagent is usually a Markdown file whose header gives a name, description, allowed tools and model, and whose body is its system prompt, stored in the project or personal configuration folder. Subagents can spawn subagents up to a limit, and two to three levels is the sweet spot; a subagent needs the delegate tool itself before it can spawn children.
Common built-in subagents
Use
Explorer
Read-only search of the codebase, no writing or editing; thoroughness can be set per call
Planner
Research during planning, keeping exploration out of the main window
General-purpose
Complex multi-step work that needs both exploring and acting
Explorer and planner skip the project instruction file and git status to stay fast and cheap. Ways to invoke a subagent, from weakest to strongest control:
01Name it in plain language: the agent decides whether to delegate, for this one request
02Call it with the dedicated syntax: this task always goes to it
03Run the whole session as that subagent: its prompt, tools and model
Keep in the main conversation
Hand to a subagent
Back-and-forth iteration
Verbose output you won’t need again
Stages that share context
Work that needs restricted tools or permissions
Quick, targeted changes
Self-contained work that returns a summary
Latency-sensitive work
Parallel, independent work; simple jobs on a small model
Agent teams (offered by some agents, usually as an experiment) have a lead and several teammates. The lead spawns teammates, coordinates and synthesizes; each teammate is a separate agent instance; everyone shares a task list and claims items from it; each agent has a mailbox, so teammates talk directly. Teammates don’t inherit the conversation but load the project instruction file and skills. Teams suit reviews from several angles, independent modules, debugging with competing hypotheses and work that crosses front and back end; sequential tasks, edits to the same file or dependency-heavy work belong in one session or with subagents.
Subagents
Agent team
Communication
Report back to the caller only
Teammates talk directly
Coordination
The lead manages
Shared task list
Best for
Tasks where only the result matters
Work that needs discussion
Token cost
Lower
Higher
Dynamic workflows use a script running in the background to orchestrate many subagents, for codebase-wide bug hunts, migrations of 500 files, research that needs cross-checking and planning from several angles.
Long runs. One form is a goal mode: you hand over a verifiable end state, such as “the lint command exits with zero errors”, and an evaluator rechecks after every attempt, stopping when the condition holds or the maximum number of attempts runs out. “All integration tests pass” works as a goal; “refactor the code” does not, because nothing can check automatically whether it is done. The other is a timed loop that repeats one prompt at an interval, such as “check CI and fix whatever broke” every 5 minutes, until you stop it.
Case three: 16 agents build a C compiler
16Parallel agents, about 2,000 sessions over two weeks
~100KLines of Rust that build Linux 6.9 on x86, ARM and RISC-V
2B / 140MInput / output tokens; just under $20,000 in API cost
~99%Pass rate on the major test suites
Plain infrastructure did the coordination:
Each agent had its own container and copy of the code.
Tasks were claimed with lock files in one folder, so no two agents did the same job.
After implementing and testing, an agent pulled everyone else’s work, then merged, pushed and released its lock.
New sessions read a progress document to regain context.
Specialist agents handled documentation, performance, quality and deduplication.
Automated tests showed how far the work had come.
There was no chat and no manager agent. The limits: merge conflicts were frequent; once every agent hit the same Linux kernel failure, parallelism stopped helping; part of 16-bit x86 still relied on GCC; and the generated machine code was less efficient than GCC’s.
Programming 3.0 and uncertainty
One way to divide it
What humans write
Through
Becomes
Programming 1.0
Machine instructions or assembly
Run by hardware
Programming 2.0
A programming language
Compiler
Machine instructions
Programming 3.0
The goal, stated in natural language
Agent
A programming language, then machine instructions
Each step up raises productivity and uncertainty together. Karpathy’s Software 3.0 divides it differently: 1.0 is source code, 2.0 is neural network weights (collect data, train), 3.0 is prompts to an LLM (natural language). Judging whether a job should be done the 1.0, 2.0 or 3.0 way is an important engineering skill.
Uncertainty is not new. Hardware (32 vs 64 bit, byte order), environment (concurrency, unreliable networks) and software (compiler versions, dependencies that update quietly) were handled in 2.0 by fixed-width types, memory consistency models, APIs that are safe to retry and full testing before upgrades. 3.0 adds more: non-determinism in GPU computation and floating-point variance, code generation that changes the environment as it writes, randomness in sampling, and the vagueness of a one-line prompt. Uncertainty sits in five layers: natural language (the meaning may be misread); retrieval and context (results change, indexes update, content gets truncated); model generation (sampling, software and hardware non-determinism); the agent and tool loop (tools fail, the environment changes); and the generated program (usually deterministic once written, unless it calls models or tools at runtime).
01Specify: state assumptions and have the model confirm unclear points before acting; hand over outcomes rather than a prompt, meaning requirements, invariants (conditions that must always hold), constraints and acceptance tests
02Control: separate proposing from executing and verify a plan before committing; manage context deliberately, owning memory, sources and retrieval, and expire stale information
03Verify and govern: tighten the generate-and-check loop, since frequent cheap checks beat one expensive review at the end; measure one layer deeper with tests, several metrics and simulated workloads you can reason about
Don’t expect the agent to be deterministic; make the system around it reliable, and match autonomy to risk.
Evidence from the field: Codex and OpenAI
Tasks given to Codex grew sharply in six months
Share of users running tasks of this size, %, Dec 2025 vs May 2026; person-hours are the hours a person would need
≥ 1 person-hour · 2025.1235.4%
≥ 1 person-hour · 2026.0570.2%
≥ 8 person-hours · 2025.122.1%
≥ 8 person-hours · 2026.0525.6%
OpenAI Codex report (June 2026)
Users running full-day tasks grew 12×. Active users grew more than 5× in the first half of 2026; under 1% of individuals and 17.3% of organizations used Codex in the past 28 days, while nearly every OpenAI employee did.
28.6%OpenAI users running more than 5 agents in a week
~2.5Hours of agent time per day for the median employee
~71Hours per day at the 99th percentile, up 88% since April
3.14×Agent workdays per human workday in research, September 2026
Before June 2026, agent run time was smaller than human work; after, it was larger. Most researchers run four or more parallel workstreams, and lines changed per engineer are close to 8× the pre-2025 average. The human role shifts to planning, delegating, monitoring, reviewing and integrating.
The longer the task, the less often it succeeds without help
Share of tasks succeeding without human intervention, %, by the time a person would need
86%<15m
76%15–30m
69%30m–1h
61%1–2h
57%2–4h
43%4–8h
40%8–16h
23%16–32h
13%32–64h
16%64–128h
OpenAI report (September 2026)
The delegation ladder, bottom to top: chatbot (AI answers, the human does the work), assistant (human and AI work together), agent (the human hands over a whole unit of work), multi-agent worker (the human coordinates several agents), systematized workflow (skills and agents run the process). Measured on this ladder, OpenAI’s workforce as a whole is still on the lower rungs.
Putting it to work
01Get one agent working end to end before you go parallel. Run in parallel only tasks that are independent (different modules, different bugs), one worktree each; tasks that touch the same file should not run in parallel.
02Hand verbose searching, log reading and research to a subagent and have it bring back only a summary; give it the fewest tools it needs, read-only if reading is enough.
03Write every long task as an end state that can be checked automatically: a command plus the expected result, such as “the test command exits with 0” or “all 12 months in the report have figures”. If you cannot say how to check it, don’t hand it over yet.
04The longer the task, the likelier it fails. Split anything a person would need several hours for into smaller tasks that can each be accepted, and hand them over and check them one at a time.
05Break the workflows you repeat into small skills, each with entry conditions, forbidden actions and the next step; put human approval on steps that change production data or send anything outside.
06When tool output runs long, borrow Caveman’s rules so the agent reads only what matters: errors and the first and last lines of logs, changed lines of diffs, keys and structure of JSON.
One vendor’s models only: Antigravity CLI, Grok Build; any model service: OpenCode with 75+ providers, Kilo with 500+ models
Sandbox vs permission prompts
Operating-system sandbox: Codex CLI, Copilot CLI, OpenHands in containers; per-action permission prompts: most tools
Local vs cloud
Local-first, no usage reporting: Goose, Nanocoder, gptme; hand off to the cloud: Copilot CLI, Codex, Mistral Vibe
Kitchen sink vs minimal
Most features: oh-my-pi; four tools only: Pi, mini-SWE-agent (about 100 lines of Python)
Self-check
Should this kind of work stay in the main conversation or go to a subagent?
Back-and-forth iteration
Verbose output you won’t need again
Quick, targeted changes
Work that needs restricted tools or permissions
Latency-sensitive work
Parallel, independent work
A subagent starts from zero, cannot see the earlier conversation and spends time gathering context; it suits self-contained work that returns a summary.
Self-check
Handing a long task to a goal mode, which of these works as the goal?
The goal must be a verifiable end state that an evaluator can check automatically after every attempt.
04CHAPTER 4
Agent design I: the loop, tools and context智能体设计 I：循环、工具与上下文
Key points
Agent = model × harness; changing only the harness raised the tasks one model solved on two SWE-bench versions by 1.7 to 2.3×.
Designing tools is designing context: few tools that don’t overlap, output that saves tokens, errors that tell the model how to fix things.
Context costs money, time and attention, and the model is most likely to miss an answer placed in the middle.
When the window gets too large, apply admission, offloading and transformation in turn; having the LLM summarize is the last resort.
ReAct and the agent loop
ReAct (reason plus act) puts reasoning and acting into one loop: think (what is known, what is missing, what to do next), act (issue a structured call the environment can run, such as “search for this function name”), observe (the facts the environment returns). Reasoning alone has nothing to check it, so small errors compound into confident hallucination; acting alone has no plan behind the calls, so the model goes round in circles. ReAct lets thinking steer the next action and observation correct the next thought, until the observations answer the question or a stop rule ends the run.
Written as code, a minimal agent loop is:
01Start the message list with the system prompt and the user request
02Loop up to a step limit: call the model and append its reply
03If the model gives a final answer, return it
04Otherwise, for each tool call: check the arguments, run the tool, turn the result into text and append it as a tool message
05Past the step limit, exit with an error
As the designer of a harness you do four things: write tool definitions (which tools exist and what arguments each takes); assemble the context; check requests, verify permission and execute; and optionally post-process results and manage context. A coding agent’s three parts must be designed together because they share one context window: the model plus the loop; tools and guardrails (permissions, sandbox, approvals); and state (what is in the window now, and what carries from one step to the next).
Tool design: search and read
Approach
How it finds code
Fits when
Main cost
glob / grep
Matches file-name patterns or literal text
The name is known or guessable
Each search costs a turn and context; literal matching misses code phrased differently
Semantic index
Turns code chunks into vectors that capture meaning (embeddings) and searches by meaning; or, as Aider does, a repo map (the file tree plus functions and classes ranked by importance)
Your words don’t match the code’s
Building the index and keeping it in sync
Outside clues
Stack traces, failing tests, git history
A report already points at the code
Present in only some tasks
Language server (LSP)
The compiler’s view of symbols: where defined, who references them, what type
The change must follow references, as in a rename
One resident server per language, slow to start
In practice, start with grep and add an index or LSP when names are not enough.
Tool design: edit
Approach
How it works
Risk
Whole-file rewrite
The model regenerates the entire file
Output grows with file length; code you meant to leave alone quietly changes in long files
Unified diff / patch
The model emits a patch and a tool applies it
Stale line numbers or surrounding code make a change block (hunk) fail to apply
Search and replace
An exact anchor string from the original plus its replacement
Fails if the anchor is not unique, is stale or has mismatched whitespace
Structured / AST edit
Operations on symbols in the syntax tree (AST, a structured form of code), such as rename or add a parameter
Needs tooling per language and can express fewer edits, but cannot produce invalid syntax by construction
Tool design: execution and the rest
The shell (command line) is the most powerful and most dangerous tool. One call can chain, filter, loop and retry; it reaches the file system, network, package managers and any program; files and processes it leaves behind outlive the call. Whatever the shell can do, the agent can do. Think through isolation (where it runs, what it can reach), failure signals (errors the agent can understand and act on), output (enough but not too much), duration (long-running and background processes) and budget (after how many retries to stop and hand back).
Five more kinds of tools are common: memory and notes (so the plan survives compaction); subagents (explore and return a shortlist, not the transcript of 40 file reads); web search and fetch (read the installed source first, then the web); browsers and visual feedback (send screenshots back to the model); and asking the user (one cheap question beats a confident wrong patch).
Few and non-overlappingOverlapping tools make the model hesitate over which to use
Specific, not generalsearch_orders(customer_id) beats a raw run_sql that executes any SQL
Token-efficient outputTruncate, paginate, summarize; designing tools is designing context
Errors that teachReturn a hint on how to fix it, not just “invalid argument”
Idempotent by defaultRunning the same call twice has the same effect as once, so retries are safe
An interface built for agents solves more than a bare shell
SWE-agent ablation (removing one design choice at a time to see the effect), GPT-4, resolve rate, %
Full SWE-agent interface18%
Full interaction history15%
Whole-file viewer12.7%
Shell only11%
No dedicated editor10.3%
SWE-agent ablation, original lecture notes
SWE-bench tests agents on real GitHub issues: given a codebase and an issue description, does the agent’s patch make the tests pass?
From prompt engineering to goal engineering
Era
Name
Question
Focus
2023–2024
Prompt engineering
How do I instruct the model?
Wording, examples, output format
2025–2026
Context engineering
What should the model see?
Retrieval, memory, history, compaction
Now
Harness engineering
How should the agent run?
Loop, tools, sandbox, retries, stopping
Next
Goal engineering
What counts as success?
Objectives, criteria, constraints, trade-offs
Prompting technique
How
When to use it
When to move on
Zero-shot
Instruction only
Common task, format easy to describe
Rules hard to state: go few-shot
Few-shot
Instruction plus a few examples
Unusual output shape that is easier to show than describe
Needs steps: go chain-of-thought
Role and persona
Assign a role
Tone or judgment matters, specific audience
Needs steps: go chain-of-thought
Chain-of-thought
Write the reasoning steps, then the answer
The task really has steps and errors come from skipping them
Several paths: go tree-of-thought
Tree-of-thought
Branch into lines of thought, score each, prune the weak
Several routes are viable and scoring is cheap
One right answer: go self-consistency
Self-consistency
Answer several times, then vote
One answer and costly mistakes; send split votes (such as 3/5) to a human
Stages separate: go prompt chaining
Prompt chaining
One job per call, checked at each handoff
Staged work where each stage can be checked; retry only the stage that fails
Tuning the prompt itself: go meta-prompting
Meta-prompting
Have the model rewrite its own prompt
You have a scored evaluation set
Gains plateau: revisit the choice
Four general rules: start simple and add context as needed; separate the parts (context, instructions, tools, output spec, ordered as instructions, delimiter, data); be specific and concise, since clever phrasing does nothing; break big tasks down rather than front-loading complexity. Prompting advice differs by model, so re-tune when you switch, and some older tricks no longer help with newer models.
Context engineering: context is a finite resource
Karpathy’s definition: context engineering is the art and science of filling the context with just the right information for the next step. Context has three costs: money (every step resends the whole window), time (bigger prompts make prefill and decode slower) and attention (the fuller the window, the worse the recall, and the middle is most often ignored).
Accuracy is lowest when the answer sits in the middle
Position of the answer among 20 documents vs answer accuracy, %
76%Doc 1
61%Doc 5
55%Doc 10
59%Doc 15
71%Doc 20
Liu et al., “Lost in the Middle”, TACL 12 (2024)
Length has its own cost: in Chroma’s Context Rot tests, growing the input from 1K to 400K tokens dropped simple-lookup accuracy to 73%, and to 46% with distractors added.
An illustrative token budget for one turn: system prompt about 2K, tool definitions about 6K, project environment about 8K, retrieved material about 25K, message history about 80K and growing. When the window gets too large there are four strategies, from preventive and lossless to lossy and irreversible:
The first three strategies shrink context by 75% to 98% without summarizing
Example, tokens before and after
Admission · raw web DOM48,000
Admission · extracted result only900
Offload · 60 tool definitions42,000
Offload · index plus 3 definitions3,100
Transform · all results kept96,000
Transform · latest state only24,000
Original lecture notes, example
Strategy
How
Admission
Guard the door: targeted retrieval (just the fields, not the whole document); reshape tool output into a lean format first and truncate only when it cannot be reshaped; hand raw data to a subagent to process on its own
Offloading
Load on demand: tiered tool definitions; tool search (keep only an index and fetch a tool’s full definition, its schema, when needed); tool outputs that return a handle you can redeem for the full text; task lists on disk; persistent memory kept to a small index
Transformation
Drop history selectively: deduplicate (keep only the latest state of each object); mask old observations (remove tool results older than N steps); sliding window (keep only the start and the last few turns)
Compaction
The last resort, with an LLM: rolling summaries, a full summary at the limit, auto memory that stores key facts outside the window. Reserve room for output, keep the system prefix, and put key facts back after compacting
Context fails in five ways, and naming the failure makes the fix obvious: poisoning (a hallucination enters the window and is later cited as fact), distraction (the history is so long the agent starts imitating its own past actions), confusion (irrelevant tools and documents pull the answer off course), clash (old and new information contradict each other but both stay) and drift (the real goal is buried and swapped for a nearby subgoal).
The same model with only the harness changed solved clearly more tasks on both benchmarks: SWE-bench Verified rose from 43/169 to 72/169, SWE-bench Pro from 31/316 to 72/316. There were only four changes: compress old tool output, keep recent observations visible, detect repetition and stalls, and step in when stuck. Some of what looks like model failure is really context and control-loop failure.
Putting it to work
01When you connect a new tool to an agent (say, an orders or inventory API), check five things: does it overlap an existing tool; are its name and arguments specific; is the default output lean and paginated; does an error say how to fix it; is calling it twice safe?
02When the agent goes off track or drifts further over time, check the five context failures first: is it citing an earlier wrong conclusion (poisoning), repeating itself (distraction), led astray by irrelevant files (confusion), caught between old and new statements (clash), or has it forgotten what you wanted (drift)?
03Put the most important information at the start or the end, not buried in the middle of a long document; give the agent only the relevant passages of reference material, not whole documents.
04In a long session, have the agent save processed logs, web pages and data to a file and keep only the path, or hand them to a subagent that reads them and brings back the conclusion.
05Pick a prompting technique starting from zero-shot and move down the table as needed; give two or three examples when the format is fixed, and split staged work into a prompt chain, checking each stage before passing it on.
Further: beyond ReAct
ReAct loops every step; Reflexion loops after failure (try, evaluate, reflect, retry); plan-and-execute loops when the plan breaks (plan, execute, replan); tree search does not loop at all but expands several branches in parallel and keeps the best one.
Self-check
Which context strategy is each of these?
Take only the fields you need, not the whole document
Keep a tool index and load full definitions on demand
Keep only the latest state of each object
Have an LLM summarize the history at the limit
Hand raw data to a subagent
Remove tool results older than N steps
From admission to compaction the strategies run from preventive and lossless to lossy and irreversible.
Self-check
“The history is so long the agent starts imitating its own past actions.” Which context failure is this?
Poisoning is a hallucination cited as fact, confusion is irrelevant tools and documents pulling the answer off course, drift is the real goal replaced by a nearby subgoal.
Multiple agents are not the default; there are only four good reasons to split: specialization, isolation, parallelism and keeping context in check.
Every architecture answers three questions: who coordinates, how the task is split and how agents communicate; real systems mix several architectures.
A subtask must state its goal, boundaries, output format and completion test; parallelism pays only when the work saved exceeds the coordination overhead.
Communication between agents is a distributed-systems problem: keep control separate from content, and handle timeouts with idempotency keys and rollback paths.
Three graphs and eight architectures
Every architecture rests on three questions: the control graph (who coordinates: one supervisor, layers of supervisors or nobody; is control handed over mid-run), the task graph (how the work is split: a planner splits it up front, agents bid for it, or it emerges from local rules) and the communication graph (direct messages, broadcast or a shared workspace).
Class
Architecture
Core idea
Fits
Weakness
Centralized
Supervisor and workers
One planner, many executors
Subtasks that split and run in parallel
The supervisor becomes the bottleneck and its context grows fastest
Centralized
Hierarchical
Supervisors manage sub-supervisors
Tasks that break down level by level
Errors amplify down the levels; hard to decide how much context crosses each level
Centralized
Pipeline / DAG
Each stage feeds the next, with an agreed data format at each handoff
Clear stages in a fixed order
Errors flow downstream with no going back; easiest to debug
Decentralized
Market and auction
Agents bid on capability, cost and speed
Dynamic allocation; work lands with the cheapest qualified agent
Bids are self-reported and can overstate ability
Decentralized
Blackboard
Coordinate through shared state (a database, file system or git repository)
Many agents working on one shared artifact
Stale state; consistency, concurrency, provenance, versions and cleanup all need managing
Decentralized
Committee and debate
Answer independently, then a judge or a vote decides
Tasks that need judgment
Groupthink: same model, same context, same blind spots
Decentralized
Swarm
No fixed coordinator; agents negotiate directly
Local negotiation
Messages grow with the square of the agent count (about n² for n agents), conflicts are hard to resolve, rare in products
Hybrid
Dynamic topology
Add, remove or rewire agents mid-run
Task structure unknown in advance; keep simple tasks cheap
A different shape every run, so failures are hard to reproduce
A DAG (directed acyclic graph) is a task dependency map that only moves forward and never loops back: B waits for A, while C and D can run at the same time.
Real systems are hybrids. For example, a planner agent sends work along a task DAG it can adjust at any time to coder, researcher, tester and reviewer agents, all sharing one git repository, and replans when needed. That uses supervisor, hierarchy, shared memory and pipeline at once.
01Planner agent
→
02Task DAG, adjustable at any time
→
03Coder · researcher · tester · reviewer
→
04Shared git repository
→
05Replan when needed
Task decomposition
There are six ways to split work: by data (each agent takes a slice of the documents), by expertise (security, performance, correctness, legal), by stage (plan, build, test, review), by hypothesis (each agent investigates one possible explanation), by tool (web, database, code execution) and by permission (different administrative domains). To review a pull request that touches 200 files, for instance, split by file first, then by security and performance.
A good subtask states: the goal; inputs and relevant context; boundaries and exclusions; allowed tools and permissions; the output format (a schema listing the fields to return and their types); the completion test; budget and deadline; and which other tasks it depends on.
A bad subtask
“Research storage systems”
A good subtask
“Find storage-related papers published in 2026; return title, date, systems contribution, evaluation scale and a link to the primary source; exclude database query optimization”
Granularity is a trade-off: too coarse and there is no parallelism; too fine and scheduling and communication dominate. Total multi-agent time is roughly planning time plus critical-path time (the longest chain of dependencies) plus communication time plus merge time. Parallelism helps only when the serial work it saves exceeds those costs. Rather than sending 20 agents one-line lookups each, merge them into 4 subtasks.
Decomposition can be static (known before running, like summarizing 5 PDFs), dynamic (spawn a lookup when a dead link turns up), recursive (a reviewer spawns a subagent for each subdomain) or opportunistic (an idle agent claims the next unlabeled document). Dynamic decomposition needs a spawn limit, a budget, a depth limit and a stop condition, or delegation recurses without end.
Common decomposition failure
Example
Overlap
Two agents summarize the same paper
Gap
One takes “2020–2023”, another “2025–2026”, and 2024 is missed
Hidden dependency
Pricing needs a size estimate, discovered only at runtime
Unmergeable outputs
One returns free text, another JSON
Lost global context
Every section is fine on its own, yet the report contradicts itself
Straggler
Six agents finish in 20 seconds, one takes 4 minutes and everyone waits
Communication
Dimension
Options
Rule of thumb
Topology (who can talk to whom)
Supervisor and workers, peer to peer, mesh
Follow the task’s dependencies
Medium (how information is exchanged)
Message passing (queues or a bus), shared memory (such as a shared task list), files and artifacts (including databases)
One medium per link
Channel (how it is delivered)
Point to point, broadcast, publish and subscribe, shared blackboard, request and response, artifact handoff
Broadcast only when everyone must know
Format and meaning
Natural language plus control fields
Route on fields, never on the wording
Natural language is flexible but vague; structured messages are easy to validate but less expressive. In practice, use both: messages carry fields such as task_id, type, status, claim, confidence, evidence and artifact. Keep control separate from content with a small set of operations the runtime understands: DELEGATE, QUERY, REPORT, HANDOFF, UPDATE, CANCEL, ACK (received), ERROR and COMPLETE. The runtime can then route, retry and stop without asking an LLM what a message means.
Communication is a hard distributed-systems problem. Agent A asks B to “buy 8 GPU instances”, and the network times out with no reply. Did B do it? Retry or give up? Guarantee “at most once” (might skip it) or “at least once” (might do it twice)? Can you get “exactly once”? Use the task_id as an idempotency key (one ID takes effect only once, so resending cannot buy twice), prepare compensation and rollback paths, and decide in advance who answers for a duplicate purchase.
Multi-agent systems inherit every hard problem of distributed systems: concurrency and non-determinism; unreliable networks (delay, loss, reordering, duplication); no shared clock; partial failure (you cannot tell a slow agent from a dead one); keeping copies of state consistent; and debugging and observability (logs are hard to line up when machine clocks drift). On top of that come scheduling (who runs next, and for how long), resource allocation (models, CPUs) and state management (checkpoints, memory, provenance, versions), with no request flow planned in advance. LLMs are black boxes, which makes it harder:
Problem
What it looks like
Information loss
A summary drops what the worker knew
Context pollution
Key information gets buried
Stale information
X has changed to 5, but B still acts on X=3
Conflicting beliefs
A says safe, B says vulnerable
Nobody owns it
Every agent thinks another is responsible
Communication loops
A asks B, B asks C, C asks A
Hard to diagnose
Nobody can say why it failed
A2A and MCP
A2A (agent-to-agent) is a protocol for agents to talk to each other. Every A2A server publishes an Agent Card at /.well-known/agent-card.json with its identity, skills, address, capabilities (such as streaming) and authentication. There are three basic objects: messages (one exchange between client and agent), tasks (a piece of work with an ID and a lifecycle) and artifacts (what a task produces). Follow-up messages join the same task. A task is submitted, then working; it can pause for input or authentication and resume; it ends completed, failed, rejected or canceled. Long tasks can be followed by polling (short tasks), streaming (watch output as it comes) or push notification (tasks of several hours). A2A standardizes only communication, not orchestration: who delegates to whom, how work is split and when to stop are still the harness’s decisions.
01Submitted
→
02Working
→
03Pauses for input or authentication, then resumes
→
04Completed · failed · rejected · canceled
MCP
A2A
Main object
Capabilities / tools
Agents / tasks
Connects
An agent to a service
An agent to an agent
What you discover
Tools, resources, prompts
The other agent’s capabilities
One piece of work
One tool call
A long-running task
Internal state
Exposes a capability interface
Best kept private
Analogy
A device or API interface
A remote call between services (RPC)
One A2A remote agent may call twenty MCP servers inside. With MCP, the host, not the model, decides what is exposed.
Design principles
01First ask whether you need multiple agents at all: when one agent is enough, more usually just add overhead
02Let the task’s dependencies set the topology, and keep communication to a minimum
03Find where information is lost: whichever node summarizes is where detail disappears
04Draw failure-isolation boundaries so one stuck part cannot stall the whole system
A dense mesh is rarely a good default. More communication adds context pressure without necessarily adding useful information.
Putting it to work
01Try a single agent before splitting. Split only if one of these holds: the context will not fit; parts need different permissions or tools; subtasks are independent and parallelism clearly saves time; or different expertise must check different aspects.
02Write a task card for each subtask: goal, inputs, what not to do, allowed tools, return format (list the fields), what counts as done, budget and deadline. A shared return format is what makes the final merge possible.
03After splitting, check for the six failures: overlapping scopes; gaps (time periods, regions and product lines are the easiest to miss); unstated dependencies; outputs that cannot be merged; overall consistency; and one subtask so slow it should be split again.
04At the merge, have one person or one agent read the whole thing specifically to find contradictions between parts, because summarizing is where detail gets lost.
05Give every action that spends money, sends a message or changes data a unique task number; check whether that number has already run before acting, and after a timeout query the status rather than retrying blindly.
Self-check
Which architecture fits each kind of task?
Independent subtasks that split and run in parallel
Clear stages in a fixed order
Tasks that need judgment
Assigning work dynamically to the cheapest qualified agent
Self-check
One agent covers “2020–2023” and another “2025–2026”, and 2024 is missed. Which decomposition failure is this?
A space left between scopes is a gap; overlap is two agents doing the same thing, a hidden dependency is an ordering discovered at runtime, and a straggler is one unusually slow subtask.
06CHAPTER 6
Agent design III: memory and reliability智能体设计 III：记忆与可靠性
Key points
Every stage of memory is a trade-off: whether to write, what form to store, where to keep it, when to retrieve, how to generalize, when to forget.
Every memory needs a validity time and a source, and forgetting is as essential an operation as remembering.
At 95% per-step success, only 8% of 50-step tasks finish without an error; errors compound and also flow downstream.
Failures come from five places: the task description, the harness, the model, the tools and the environment, and each has its own defenses.
Six design questions for memory
01Write or skip: is it worth keeping?
→
02Encode and organize: what form should it take?
→
03Store: where should it live?
→
04Retrieve: does this task need it?
→
05Reflect and generalize: what broader conclusion follows?
→
06Keep, update or delete: is it still true?
Writing. Recording every interaction misses nothing but is noisy; writing selectively is precise but can miss things. The decision to write can rest on an explicit user request, a fixed rule, a small classifier or a periodic LLM review.
Form
Example
Trait
Raw event
“We moved serving from vLLM to SGLang.”
Faithful but large and redundant
Unstructured summary
“Uses SGLang for model serving.”
Compact, may lose context
Structured state
serving_engine = SGLang
Queryable, but bound by the field definitions
Knowledge graph
Project → USES → SGLang
Connects entities and can express change over time
Store
Examples
Good at
Plain files
The project instruction file, scratchpads
Easy for people to inspect, diff and search
Key-value or relational database
Redis, SQLite, Postgres JSONB
Exact lookup, filtering, transactions
Vector store
pgvector, Chroma, Pinecone
Finding content with similar meaning
Graph database
Neo4j, FalkorDB (Graphiti)
Querying relations and following them several hops
When memory is used
How
Pros
Cons
Automatic
Search, call the LLM and write every turn
Predictable, simple
More latency and irrelevant context
Runtime-controlled
Search or write only when a condition holds
Selective, efficient
Needs rules or a trained classifier
Model-controlled
Give the LLM search, add, update and forget as tools
On demand
May fail to read or write when it should
Many systems mix these, and reads and writes can follow different policies.
Retrieval. The basic pipeline is query, vector search, take the top k most similar, put them in the prompt. But similar meaning is only one signal. Also weigh scope (the same user, project, entity or task), importance (the cost of missing it or bringing in the wrong thing), validity (does it still hold now, for this task) and provenance (source, confidence, evidence).
Reflection. Repeated questions about vLLM, SGLang, TRT-LLM and LLM scheduling generalize to “sustained interest in LLM inference systems, medium confidence”. Generalize once enough new evidence has built up (every N events, or when the context fills) or when facts conflict or repeat, preferably as a background job.
Forgetting matters as much as remembering. Reasons to forget: staleness, errors or contradictions, privacy and security, storage and retrieval cost. Ways to do it: mark invalid, delete permanently, expire automatically (TTL), or tier the store and evict from active memory. Every memory needs a validity time and a source: “serves with vLLM” recorded in April 2026 is replaced in June by “serves with SGLang”, noting that it came from conversation 28.
Who owns it. Either the agent does (files in the harness: the project instruction file, scratchpads, task notes, saved by the harness) or a dedicated memory system does (a service the agent calls, making memory an interface shared across agents and sessions).
Memory system
Kind
How it stores and retrieves
Mem0
Fact-centric
Scoped by user, agent or run; vector search plus filters, optional graph memory
Graphiti
Knowledge graph with time
Extracts entities and facts with validity periods from conversation snippets; mixes meaning, keyword and graph search
supermemory
Profiles, memories and documents
Maintains user profiles, memories and documents together, linked to source and time
Memory failure
What it looks like
Retrieval miss
The relevant memory exists but is not found
Context pollution
Too many irrelevant memories enter context
Memory hallucination
An inference is stored as fact: “studies at University A” becomes “visiting student at University A”
Staleness
An old fact overrides new information
Isolation failure
One user’s memory reaches another user
Over-generalization
A one-off event becomes a lasting preference
Persistent prompt injection
Untrusted content is stored and later comes back as an instruction
Unbounded growth
The store keeps growing, and retrieval gets costlier and less accurate
Evaluate memory on four fronts: retrieval (precise and complete?), writing and updating (precise and complete? how much stale memory?), governance (deletions honored, scopes isolated?) and the system (latency, extra tokens, storage growth). End-to-end benchmarks mainly test whether an agent can recall and reason over a long conversation history, covering recall, temporal reasoning, updates, preferences and declining when it should: LoCoMo (ACL 2024, 10 long conversations, 1,540 questions), LongMemEval (ICLR 2025, five memory abilities, 500 questions) and BEAM (ICLR 2026, up to 10M tokens, 2,000 questions). A memory stack comes in two shapes: a separate memory layer (the agent runtime, plus memory management such as Mem0 or Letta, plus a vector, SQL, graph or object store), or memory inside the harness (managed by the runtime itself and stored as Markdown files).
Reliability: errors compound
At 95% per step, only 8% of 50-step tasks finish without an error
Probability of an error-free task, %, by number of steps, assuming independent steps
95%1 step
77%5 steps
60%10 steps
36%20 steps
8%50 steps
Computed as 0.95 to the power n
A task succeeds only if every step succeeds, and an early error carries into later steps, so real runs do worse than this estimate. Failures can start in five places: the task description, the harness, the model, the tool layer and the environment.
Failure
Example
Fix
Vague goal
“Fix the login bug”, and it fixes the wrong one of three
State the file, the symptom and how to reproduce it
Proxy goal
“Make the tests pass”, and it deletes the failing test
Forbid editing tests; review the changes
Unstated constraint
“Speed up search”, and it breaks the public API
Say what must not change
No definition of done
“Tidy up the parser”, and it refactors forever
Give an acceptance test and a stop rule
Conflicting goals
“Smallest change, cover every edge case”, and it quietly drops the edge cases
Rank the goals explicitly
Failure
Example
Fix
Context truncation
The earliest turns are cut and it forgets “keep the API unchanged”
Pin key rules somewhere that is never cut
Memory error
Treats a stale note as fact and edits a file that has moved
Date memories and verify before use
Corrupted state
A half-edited file is left behind and later steps build on it
Checkpoints; roll back failed steps
Runaway loop
No step or cost limit, 200 steps with no progress
Cap steps, cost and time
Blocked fan-out
Waits on every subagent and one slow one holds up everything
Set timeouts; use the partial results that have arrived
Failure
Example
Fix
Hallucination
Calls a requests.fetch() that does not exist
Check calls against the real documentation
Instruction violation
Changes a function signature it was told to keep
Restate the constraint; check the changes
Faulty reasoning
Assumes a timeout means the network is down and adds retries to a deadlock
Test the assumption before fixing
Says one thing, does another
Says it will change only parser.py, then edits three other files
Check every action against the plan
Unverified success
“The tests should pass now”, announced without running anything
Require test output as evidence
Long-horizon decay
Undoes its own fix from step 12
Reread the current state; split into short tasks
Failure
Example
Fix
Truncated output
Returns the first 100 lines of a log and misses the error on line 4000
Return errors first; support paging
Silent failure
Exit code 0 and an empty result, read as “no matches”
Separate “empty result” from “failed”
Vague format
A date without a time zone puts the task 8 hours off
Make field types explicit
Transient failure
Decides the service is down after one timeout
Retry a few times with a pause before concluding
Misleading error
A typo reported as “permission denied”, so it retries with sudo
Return precise errors you can act on
Failure
Example
Fix
Stale state
Yesterday’s build cache lets old code pass the tests
Clear the cache and rebuild
Version mismatch
Python 3.8 instead of 3.12, and the code crashes in production
Pin dependency versions with a lock file
Random failure
Fails in leap years
Rerun failing cases and quarantine the flaky ones
Page redesign
The checkout page changes and it clicks the wrong button
Reread the page before every action
Rate limiting
HTTP 429 after 50 calls stops a migration halfway
Throttle calls; checkpoint long tasks
Task description: write “done” as a testGive acceptance tests up front and list what must not change
Harness: bound every runCap steps, cost and time; checkpoints you can roll back to
Model: demand evidenceEvidence for every claim, verification, a stronger model when needed
Tools: fail loudlyExplicit field types; errors kept separate from empty results
Environment: pin and isolateA fresh sandbox with fixed versions every time; retry transient failures before judging
The field still knows little about agents running at scale, and written best practice is thin.
Putting it to work
01Before handing over a task, answer five questions: is the goal specific (which file, what symptom); could it be gamed (say, by deleting tests); what must not change; what counts as done; which goal wins when goals conflict? Write the answers into the task.
02Give every run three limits: maximum steps, maximum spend and maximum time; set checkpoints where you can roll back, and return to the last good state on failure.
03Make the agent argue from evidence: “tests pass” comes with the test output, “fixed” with before-and-after results from the reproduction steps.
04Set rules for memory: every entry carries a date and a source; conclusions the agent inferred are marked as inferred; anything personal gets an expiry date; clear out stale and contradictory entries now and then.
05The more steps a task has, the likelier it fails, so split where you can: several segments, each accepted on its own, are far more reliable than one long unbroken chain.
06When diagnosing a failure, go through the five sources in the tabs above: first whether the task was ambiguous, then the harness’s limits and state, then the model’s behavior, and finally tools and environment.
Self-check
Where does each failure come from?
Told to “make the tests pass”, it deletes the failing test
No step or cost limit, 200 steps with no progress
Says “the tests should pass” without running anything
A date without a time zone puts the task 8 hours off
Fails in leap years
Self-check
At 95% success per step, with independent steps, what is the chance a 20-step task finishes error-free?
0.95 to the 20th power is about 36%; 5 steps give about 77%, 10 steps about 60%, 50 steps about 8%.
07CHAPTER 7
Agent design IV: evaluation智能体设计 IV：评估
Key points
What gets evaluated has moved from a single model to the whole of model, prompt, tools and harness, and scoring now rests on tests and the final state of the environment.
Eight representative benchmarks each test one environment: question answering, function calls, the web, the desktop, a code repository, customer service, the terminal and a simulated company.
It is hard for three reasons: leakage has four routes; exams that give the full instruction up front reward guessing; and success rate alone ignores cost and safety.
Benchmarks get maxed out ever faster; judging a real agent means weighing capability, cost and speed together.
A benchmark is a public set of standard test questions, so everyone can compare models or agents on the same exam.
History and what is evaluated
There are two ways to evaluate an agent: by component (planning and task breakdown; extracting, consolidating and retrieving memory; discovering, choosing and filling in tools; recovering from errors) or end to end (success rate, cost per task, time to finish). Benchmarks are moving toward larger scope, less specific instructions, more realistic settings and fresher questions.
MMLU-Pro, LiveBench, LiveCodeBench, FrontierMath, Humanity’s Last Exam
Difficulty, robustness, fresh questions
2025–2026
GDPval, BrowseComp, Terminal-Bench, ARC-AGI-3
Larger scope, closer to real use
What is evaluated
Representative benchmarks
What success means
Predicting text
PTB, WikiText
High probability for text that actually occurs
Giving the right answer
MMLU, GSM8K, GPQA, HLE
Matches the reference answer
Giving a preferred response
Arena, AlpacaEval, MT-Bench
Rated well by people or a judge model
Writing code that runs
HumanEval, LiveCodeBench, SWE-bench
Passes executable tests
Completing interactive tasks
WebArena, OSWorld, τ-bench, Terminal-Bench
The environment reaches the required state
On metrics: early work used perplexity (how “surprised” a model is by text it has not seen; lower is better), accuracy or F1, and preference win rate. Capability is measured with pass@k (at least one of k runs passes) and pass^k (all k runs pass, introduced by τ-bench). Cost is mostly LLM spend; tool calls and the extra cost of cache misses are often overlooked. Latency is not just the model’s latency either.
A benchmark run: for each task, reset the environment to its initial state; while the task is not done and the budget is not spent, observe the environment, let the agent pick an action from the instruction, observation, history and tools, and execute it; finally an evaluator scores the final answer, the final environment state and the whole trajectory.
Eight representative benchmarks
Benchmark
Where the agent works
Typical actions
Scored on
Notes
GAIA
Web, files, code tools
Search, inspect files, compute
A short final answer
466 human-written questions at 3 levels (up to 5 steps, 5 to 10 steps, any length); scores the whole agent system; does not test actions like sending email or booking tickets
BFCL
Function and API interfaces
Structured function calls
Calls, arguments and resulting state
Checked by syntax-tree comparison or by execution; common errors are invented IDs (it should ask the user) and unnecessary calls; read the category scores, not the total
WebArena
Self-hosted websites
Click, type, navigate
Website or backend state
812 tasks across shopping, admin, Reddit, GitLab, maps and a wiki; the agent sees the page’s accessibility tree (a structured list of page elements); the Verified version drops the LLM judge for deterministic checks on recorded HTTP requests
OSWorld
A full desktop virtual machine
Mouse, keyboard, application actions
File, application and system state
369 tasks; the agent sees screenshots; OSWorld 2.0 (2026) has 108 long workflows, often over an hour, with about 27 checkpoints for partial credit
SWE-bench
A software repository
View, edit, run tests
Whether the patch passes the tests
Variants: Verified (human-checked), Lite (300 problems), Multilingual (9 languages), Multimodal, Pro (harder, multi-file)
τ-bench
A customer-service API and dialogue
Talk with the user, call APIs
Database state and replies
The user is played by an LLM and the agent must follow an airline’s policy; reward is the action score times the reply score; read-only calls may differ from the reference, but one extra write is a failure
Terminal-Bench
An isolated terminal container
Any shell command
Task-specific verification tests
Programming, operations, data processing, model training, security and scientific workflows; TB 2.0 has 89 tasks, TB 4.0 has 66 after dropping maxed-out ones, with an 8-hour timeout per task
TheAgentCompany
A simulated software company
Use web apps, write code, talk with colleagues
Process checkpoints and final state
175 tasks across software, project management, data science, HR and finance; partial credit = 0.5 × share of points + 0.5 × whether fully completed
Terminal-Bench sits between SWE-bench and OSWorld: the agent acts through a shell in a container and reads terminal output, with tasks ranging from configuration and debugging to repair and building. There is no guessing where to click, so it tests long-chain reasoning and interaction with systems. pass^k is far stricter than pass@k: it needs every one of k independent runs to succeed, while pass@k needs just one.
Why evaluation is hard
Training leakageThe question or its human fix appears in the training data
Leakage during evaluationThe agent searches the web and copies a fix published “in the future”
Environment leakageThe sandbox exposes later commits or hidden tests
Overfitting to the benchmarkPrompts and scaffolding were tuned again and again on the public test set
An agent’s performance is roughly a joint product of the LLM, prompt, scaffold, tools, retriever, memory and environment, and any of them can carry the answer in: fine-tuning used benchmark answers as example trajectories; the retrieval corpus indexed the answers; memory holds earlier attempts; researchers tuned the system by studying failures again and again. So “the model was not trained on the test set” is far from enough. Evaluating agents needs real tools, and real tools open more routes for leakage.
The full instruction up front. Most benchmarks give one complete instruction: prompt in, final state out, with no user in between. Real work is vague, gets revised and is interrupted. This format rewards confident guessing, because asking a clarifying question costs steps and earns nothing. Three things should also be tested: does it ask when the goal is unclear; does it replan when corrected mid-task; does it remember constraints across turns?
Other challenges: success rate alone is not enough (200 steps for $40 and 12 steps for $0.30 both count as “success” but are not comparable); results are random, so one run’s score is noise; nothing penalizes deleting files, sending email or spending money along the way; judging only the outcome lets lucky or unsafe runs through, while demanding the reference trajectory penalizes reasonable alternative paths.
Frontier models reach the baseline ever faster
Months from benchmark release until a frontier model reached the marked baseline
GLUE (2018) · human 87.114 months
MMLU (2020) · expert 89.839 months
GPQA Diamond (2023) · PhD expert 69.710 months
SWE-bench Verified (2024) · 33% → 95%22 months
ARC-AGI-2 (2025) · average human 6011 months
ARC-AGI-3 (2026) · 0.5% → 62.7%6 months
Original lecture notes
Benchmarks get maxed out quickly. As of September 2026, SQuAD, GLUE, SuperGLUE, MMLU, MATH, HumanEval, Chatbot Arena, MT-Bench, WebArena, SWE-bench Verified, GPQA Diamond, MMMU and OSWorld are saturated; FrontierMath, Humanity’s Last Exam, ARC-AGI-2/3, Terminal-Bench, GDPval, SWE-bench Pro, Remote Labor Index and OSWorld 2.0 are not. Saturated means the strongest models have reached the human baseline, or the top models can no longer be told apart.
Dimension
Then
The frontier now
What is evaluated
The foundation model
Model, prompt, tools and harness together
Interaction
Static, one shot
Stateful, long-running
Grader
Exact match or likelihood
Tests, environment state, people or judge models
Dataset
Fixed, public
Fresh, hidden, generated or contamination-resistant
Finishing robustly under real constraints and safety requirements
Judging real-world agents means weighing capability, usability, cost and speed together. One method ranks by Pareto dominance: A dominates B if A is no worse on every metric and better on at least one; the set nobody dominates is the Pareto frontier and ranks first.
Putting it to work
01Build a private exam from your own work: pick real tasks that have a reference answer or can be checked automatically (figures match the source table, all fields present, tests pass). Keep it unpublished and never tune prompts on it; use it only to compare versions of agents, models or prompts.
02Run each question several times and record both pass@k and pass^k. For work you need to rely on, pass^k is what counts; an occasional success means little.
03Record cost, steps, time and any out-of-bounds action (deleting files, sending outside, spending money) on every run, not just right or wrong; compare options the Pareto way, on quality, cost and speed together.
04Add a few deliberately vague questions to see whether the agent asks first, and a few that change requirements midway to see whether it replans.
05Don’t take vendors’ benchmark scores on trust: a benchmark may be saturated or leaked, so retest on your own tasks with your own exam.
Self-check
A task is run independently 4 times and succeeds once. How do pass@4 and pass^4 come out?
pass@k needs at least one of k runs to pass; pass^k, introduced by τ-bench, needs all k and is far stricter.
Cost is users × sessions × turns × steps × tokens × price; the middle three fall with shorter trajectories, the price falls with caching, batching and routing.
Cache reads make up the vast majority of tokens; any token left in context is paid for again at every later step.
Every saving has a side effect, so measure the net effect on the same set of tasks and cap every task.
Completion time is set by the critical path; harness architecture runs from monolithic to modular to microkernel as responsibility moves from code to services.
Two words first: a turn is one exchange between you and the agent; a step is one model call within a turn (usually with one tool call). A task’s LLM spend breaks into four multipliers:
01Turns per task
→
02Steps per turn
→
03Tokens per step
→
04Price per token
Fewer turns and steps
This chapter covers LLM spend only, not other compute costs.
Turn k rereads all k−1 earlier turns, so total cost grows with the square of the number of turns. Usage records from a group of coding-agent users showed that sessions ending after a single prompt made up 68% and 75% of two task sets, while a few sessions ran past 100 turns. Users come back with follow-ups because the agent had no plan, stopped early or never checked its result. The fixes: confirm the goal and plan before starting; split the task into subtasks that each carry a completion test; verify before delivering. The aim is a task finished without the user stepping in.
Most turns take only a few steps (mainly because reasoning effort was set to medium or high), though some approach 1,000. Every model call rereads the whole context, so fewer calls save money: merge independent calls into one step (read five files in one step, not five), reuse plans and skills that worked before, and read the files you need once at the start. Related research covers skill libraries (Voyager, SkillRL), distilled experience (ExpeL, ReasoningBank), cached plan templates and persistent conversation memory (Mem0, LightMem, SimpleMem). Reuse pays only when tasks recur; the cost of maintaining memory and stale entries can eat the savings.
Fewer tokens per step
97.6%Share of tokens that were cache reads in the first task set
208K / 301KMedian main-agent context per step (first / second task set)
~80KMedian subagent context per step; 90th percentile about 200K
1 : 10 : 50Typical price ratio of cache read, ordinary input and output
This is how caching saves money: for the opening part already processed in the last step, the provider has kept the intermediate results, so sending the same opening again is billed at a low cache-read price. In volume, output is about 5× smaller than new prefill written to the cache, and new prefill about 100× smaller than cache reads; context grows quickly, reaching 650K to 670K by about step 500. Any token left in context is paid for again at every later step.
Cut input
How
Drop what is stale
Truncate stale results; compact history; mask tool output already used; summarize old turns into a state
Shrink what comes in
Compress observations, keeping errors rather than whole logs; save large outputs to a file and pass only the path
Load only what is needed
Retrieve selectively, giving only relevant tools and memory; send exploration to a subagent that returns a summary
Related research: ACON cuts peak context by 26% to 54%; LLMLingua cuts tokens up to 20×, but compression pays only on long prompts; MemGPT manages context as fast memory; filtering MCP results with code takes 150K down to 2K; tool search cuts tool-definition tokens by 85%, and calling tools in batches from code cuts 37%. TRACE found that summaries cause rework.
Cut output: state length, format and what to skip in the prompt (“under 100 words”, “return only JSON”, “don’t restate the question”); set a maximum output length in the API (this can cut an answer mid-sentence, and reasoning alone can use up the limit); ask for structured output with only the needed fields (enums limit the allowed values, and any text around a tool call is pure cost); and tune reasoning effort per step (low for lookups and formatting, high for planning and debugging). Note that very strict formats can hurt reasoning.
Lower the price
Caching plus batching cuts input price to 5% of base
Price multiple relative to base input tokens, Claude API
Output tokens5×
Base input tokens1×
Batch input0.5×
Cached input0.1×
Batch plus cached input0.05×
Anthropic public pricing, original lecture notes
Batching hands requests that are not urgent to the provider to queue and process later, in return for a lower price. In practice: keep the prefix stable (leave the system prompt, tool definitions and early history alone so the cache hits), and send work that can wait through batch (evaluations, backfills, bulk jobs). Model routing takes three forms: choose up front (a router trained on preference data, which can also assign agent roles, as in RouteLLM and MasRouter); escalate on failure (a cheap model first, a larger one when a check fails; the check can be a trained scorer, self-verification or actual execution results, as in FrugalGPT, AutoMix and EcoAssistant); and re-route each step (on history and budget, with an orchestrator calling models as tools, as in ToolOrchestra). The hard part: one planning mistake by a cheap model can waste many later steps. At the systems level there is a further set of ways to lower price: MoE (chapter 9), quantization, pruning, distillation and fine-tuning for the specific workload; speculative decoding, KV-cache optimization and better scheduling; faster kernels and specialized accelerators.
Every saving has a side effect
Technique
Side effect
Compaction
Dropped files must be reread later, and a summary changes the prefix and breaks the cache
Routing
A cheap model’s bad plan adds steps or causes failure
Subagents
Each one rereads the shared files and instructions
Speculation
Wrong guesses waste tokens and compete with normal calls for the same tools
Net savings are total savings minus side effects, measured together on the same tasks. Cap every task (for example: $10, 80 model calls, 15 minutes of tool time, at most 2 retries per step). Stop when verification passes, when N steps pass with no newly passing test, when the same error repeats or when a cap is hit, and hand over with a summary. Caps can be set at twice the 95th-percentile usage of successful tasks. A task that stops with a summary is far cheaper than one that runs away.
01Same steps every time: have the LLM write a script once, then just run the script
02Fixed input format: regexes, parsers and templates cost nothing per call
03Narrow range of labels or scores: train a small classifier
04Fixed workflow with little reasoning: script the workflow and call the LLM only at the step that needs reasoning
Faster agents
Completion time is set by the critical path (the longest chain of dependencies), not the sum of every step. Faster is usually cheaper, but not always: a small model may take more steps, and parallelism spends more tokens.
Faster LLM servingEvery cost technique above also cuts latency: fewer turns, steps and tokens, faster serving
Better orchestrationAsynchronous execution (start the next step while a tool runs); persistent sessions (keep the same shell or browser open across steps); streaming and pipelining (start tool calls before the model’s reply is finished); co-locate services
Faster toolsBuilds, tests and page loads can be slower than model calls: cache tool results, move heavy computation to a server, use faster tools, set timeouts on calls that might hang; measure tool time per step before tuning the model
ParallelismIssue several independent tool calls in one step; send independent subtasks to parallel subagents and merge their summaries; shared state needs a merge step
Speculative actionsA cheap model predicts the next action and runs it while the main model thinks; a right guess saves the wait, a wrong one is thrown away at the cost of one call; best when guesses are usually right and actions can be undone
Speculation ranges from the next tool (SPORK), tool plus arguments (PASTE) and any action, up to multiple branches (B-PASTE) and whole executed chains. Predictions can come from a model (a small model drafts), self-speculation (an early branch of the same model) or history (past trajectories: PASTE learned that a search is usually followed by opening a page, so it takes the URL from the search result and opens it early). The open question: with limited GPU, CPU and API budgets, how should speculative calls be scheduled so the critical-path time saved exceeds the work wasted?
Optimization is a loop: run and record every call, token, tool and wait; measure where cost and latency go; change one thing at a time (compaction, routing, caching or parallelism); re-evaluate success, spend and 95th-percentile latency on the same tasks; repeat.
Putting it to work
01Start with the structure of the bill: if your agent shows tokens and spend per session, look at how they split. If cache reads dominate, the context is too long, so cut input first; if new writes are high, the prefix keeps changing, so stabilize the system prompt and instruction file first.
02Say everything at once: goal, constraints, acceptance criteria and files to read in one request, to avoid back-and-forth. Every extra turn rereads everything before it.
03Send unimportant exploration and log reading to a subagent or small model; use a large model with high reasoning effort for hard steps such as planning and debugging, and low effort for formatting and lookups.
04Set caps for each kind of task (money, calls, time, retries); when a cap is hit, stop and hand over a summary rather than letting it run on.
05Have the agent turn recurring, fixed-step work into a script and run the script from then on; leave only the step that truly needs judgment to the model.
06Change one thing at a time (switch on compaction, say, or a new routing rule), compare success, spend and time on the same tasks, then decide whether to keep it.
Three harness architectures
Dimension
Monolithic
Modular
Microkernel
Design principle
Everything integrated in one runtime
Abstracted behind interfaces, one controller
A minimal kernel; everything else is an outside service
Swapping components
Hard
Easy (Docker to Firecracker means changing one implementation)
Easy if the protocols match
Execution isolation
Optional
Optional
An architectural focus
Deployment
Usually together
Together or separate
Usually independent
Independent scaling
Limited
Possible
Strong
Communication overhead
Low
Low to medium
Can be higher
Typical use
A standalone agent
Agent frameworks
Multi-agent infrastructure
Strengths
Simple to build and debug, state easy to share
Clear responsibilities, testable, third parties can plug in
Services scale independently, faults stay isolated, infrastructure can be shared
Weaknesses
Tightly coupled, hard to share infrastructure
The controller still owns the whole run, and coordination logic gets complex
More ways to fail, harder end-to-end debugging, needs standard protocols
Take running a snippet of Python: the monolith runs it locally, with the fewest hops but weak isolation; the modular harness calls a uniform sandbox interface (such as SandboxProvider.run(code)), so you change the implementation without touching the agent; the microkernel sends a request to an execution service that handles allocation, isolation and the sandbox lifecycle. Responsibility moves from code to interfaces to services.
What is distinctive in seven harnesses
Harness
Position
What is distinctive
Codex CLI
OpenAI, Rust
Sandbox first, approval second: commands run in a sandbox with no network and ask for approval only to leave it; the patch format is constrained by a Lark grammar and change blocks are matched in four passes
DeepSeek Harness
TypeScript
Even the loop is a plugin, replaceable through a YAML configuration layer; each request is a frozen snapshot of the session log, which the harness only appends to; edits are compare-and-swap, so an edit fails if the file changed since the model last read it
OpenCode
TypeScript, Bun
The harness is a server, and the terminal UI, SDK and web client share one OpenAPI interface; nine replacers run from exact to anchor matching and reject overly fuzzy wide matches; a shadow git repository snapshots every step so undo can restore files
Pi
TypeScript
A minimal kernel of four tools, with no MCP, subagents or approvals; sends cache keep-warm requests only when the expected saving is at least $0.05; pi-ai supports 42 model providers and 10 protocols
oh-my-pi
TypeScript plus Rust
Hashline edits locate text by a 4-hex-digit file tag plus line number, so the model need not retype old text; one resident shell, with native search, syntax tree, LSP and debugger; snapcompact compresses old history by turning it into images
OpenClaw
TypeScript
A gateway to 27 chat channels with one queue per session; before compacting it quietly writes a dated note, and recall combines full-text and vector search; fences isolate untrusted text, reading anything from the network marks the turn as tainted, and planted sentinels watch for leaking secrets
Hermes Agent
Python
A background branch reviews recently prompted turns and then edits memory and skills; it starts compacting at 50% of the window, the earliest of these; before any action with side effects it writes the tool call to a session database
Self-check
What is the side effect of each saving?
Dropped files are reread and the summary breaks the cache
A cheap model’s bad plan adds steps or causes failure
Each one rereads the shared files and instructions
Wrong guesses waste tokens and compete for the same tools
Net savings are total savings minus side effects, measured together on the same tasks.
Self-check
On the Claude API, with batching plus caching, input tokens cost about what share of the base input price?
Batch input is 0.5×, cached input 0.1×, and the two combine to 0.05×; output tokens are 5×.
09CHAPTER 9
LLM basics: the life of a request and the transformerLLM 基础：一次请求的生命周期与 Transformer
Key points
A request passes through six stages: tokenize, schedule, prefill, decode, detokenize and stream.
Prefill is compute-bound and sets how long the first token takes; decode is bandwidth-bound and sets how many tokens per second follow.
The KV cache per token sets how many requests one GPU can serve at once; GQA brings Llama 3 70B down to 320 KB per token.
An MoE model’s memory follows its total parameters, while its compute follows only the parameters active per token.
01Tokenize
→
02Schedule
→
03Prefill: sets time to first token
→
04Decode and sample: sets tokens per second
→
05Detokenize
→
06Stream
The six stages
Tokenize. Text is split into tokens, not words; in English a token averages 3 to 4 characters, and “Serving is hard” is 4 tokens. Token count sets cost and memory, and tokenizers differ, so the same text has different token counts in different models. Byte-level BPE, the common method, starts from the 256 byte values and repeatedly merges the pair of adjacent pieces that most often occur together into a new token, so any text can be split and there are no “unknown characters”. A larger vocabulary means fewer tokens for the same text but a larger embedding table to store it.
Schedule and batch. Requests arrive one by one, but a GPU is efficient only on batches. Continuous batching lets requests join or leave at any step instead of waiting for a full batch; how many can be admitted is limited by the memory the KV cache takes.
Prefill. The whole prompt goes through the model at once, with attention computed for all prompt tokens in parallel; it is compute-bound, so longer prompts are slower and costlier; it builds the KV cache and emits the first token.
Decode and sample. Every new token after that needs a full pass through the model, and the KV cache grows by one token per step; it is bandwidth-bound, so many requests are batched together, and when memory runs short some requests are paused or queued. Sampling picks the next token from the raw scores the model gives every candidate (logits), and the settings change both the answer and how long it runs (table below).
Detokenize and stream. Each token ID is turned back into text and sent as soon as it is produced, holding back partial characters; generation ends on a stop token, the length limit or a dropped connection, and stopping early frees the cache.
Sampling setting
Example
Effect
temperature
0
At 0 it always picks the top score; above 1 more tokens get a chance and answers wander more
top-k
50
Sample only from the k most likely; k is fixed
top-p
0.9
Sample from the fewest tokens whose probabilities add up to p; the more confident the model, the fewer candidates
The transformer
01Token IDs
→
02Embedding: each ID becomes a vector of 8,192 numbers
→
03Transformer blocks × 80: self-attention plus FFN
→
04LM head: one score for every token in the vocabulary
→
05Softmax: probabilities for the next token
A model is a stack of identical transformer blocks. The input is a sequence of integer token IDs; the embedding layer turns each ID into a vector (8,192 dimensions in Llama 3 70B). The vectors pass through L blocks (80 in Llama 3 70B), each with self-attention (tokens exchange information) and a feed-forward network, the FFN (each token is transformed on its own). Finally the LM head scores every token in the vocabulary and softmax turns the scores into probabilities that add up to 1.
Every block normalizes, computes, then adds back: RMSNorm, then attention, with the result added to a trunk running through the whole model (the residual stream); then RMSNorm, the FFN, and the result added back again. RMSNorm scales values to a stable range by their root mean square, with one learnable gain per dimension; normalizing first (pre-norm) keeps very deep stacks stable in training; the residual stream runs through all 80 blocks, and each layer only adds a small update.
Self-attention lets every token read all the tokens before it. Each token produces three vectors, a query Q (what am I looking for), a key K (what do I have) and a value V (what I can contribute), by multiplying the input X by three weight matrices. The dot product of Q and K divided by √dₖ decides how much to read from each token; the division is there because dot products grow with dimension, and without scaling softmax would lock onto a single token. A causal mask fills everything above the diagonal with −∞ so a token cannot see what follows; softmax makes each row’s weights add up to 1; the output is a weighted mix of the V vectors. n tokens produce n² scores, and each layer costs about 4n²d FLOPs (floating-point operations). Storing the full score matrix S for 8,192 tokens, 64 heads and 80 layers would take about 690 GB; FlashAttention computes softmax in tiles and never stores S.
Step
Formula
Shape
Input
X
n × d
Query
Q = X W_Q
n × dₖ
Scores
S = Q Kᵀ ÷ √dₖ
n × n
Weights
A = softmax(S + M)
n × n
Output
O = A V
n × dᵥ
Multi-head attention splits Q, K and V into 64 heads of 128 dimensions, runs masked attention in each, and mixes them with Wₒ. 64 × 128 = 8,192, so the compute equals one wide head; splitting changes the patterns the model can learn (some heads watch the previous token, some the first token, some the neighborhood, some the whole text), not the cost.
The KV cache and shared KV heads
The KV cache stores the K and V of every token already read so the next step need not recompute them. Its size per token = 2 (K and V) × layers × KV heads × head dimension × bytes per number. In MHA every query head has its own KV; GQA lets several query heads share one set of KV; MQA has all query heads share one. With 8 query heads the three give 1×, a quarter and an eighth of the KV cache. Llama 3 70B’s 64 query heads share 8 sets of KV: 2 × 80 × 8 × 128 × 2 B = 320 KB per token, against 2.5 MB with MHA.
Llama 3 70B parameter
Value
Model width d
8,192
Layers L
80
Query heads H and head dimension dₖ
64 and 128
KV heads Hₖᵥ
8
FFN hidden width
28,672 (3.5 × d)
Vocabulary V
128,256
Most of Llama 3 70B’s parameters are in the FFN
Parameters, billions, by component
Attention (80 layers)12.1B
FFN (80 layers)56.4B
Embedding and LM head2.1B
Meta, The Llama 3 Herd of Models (2024), computed from the published architecture
Attention is about 151M parameters per layer and the FFN about 705M; 80 layers make 68.5B, and embedding plus LM head bring the total to 70.6B. The chart’s attention and FFN totals are the per-layer figures times 80.
Estimating from a systems view
141 GBWeights (BF16, 2 bytes per parameter)
141 GFLOPsCompute per generated token (about 2 × parameters)
320 KBKV cache per token
On 2 H200s (1,978 TFLOP/s of compute, 9.6 TB/s of bandwidth and 282 GB of memory between them), an 80,000-token prompt takes about 9.95 s of prefill and 167 GB for one request; 64 requests at once need 637 s and 1,779 GB. As context grows, attention compute grows with n² and the KV cache grows linearly with n, which is why long context is hard.
MoE: memory follows total parameters, compute follows active ones
MoE (mixture of experts) replaces the FFN in each block with a router and a set of smaller “expert” FFNs; each token goes to only a few of them (say 2 of 8), and their outputs are added by weight.
Model
Experts
Total parameters
Active per token
Weights (BF16)
Llama 2 70B
All (dense)
69B
69B
138 GB
Mixtral 8x7B
2 of 8
46.7B
12.9B
93 GB
Llama 3.1 405B
All (dense)
405B
405B
810 GB
DeepSeek-V3
9 of 257
671B
37B
1,342 GB
DeepSeek-V3 activates only 37B parameters per token
Active parameters per token, billions; compute per token (GFLOPs) is about twice this
Llama 2 70B69B
Mixtral 8x7B12.9B
Llama 3.1 405B405B
DeepSeek-V337B
Jiang et al. 2024; DeepSeek-AI 2024; Touvron et al. 2023; Llama Team 2024
Less compute at similar quality: Mixtral matches Llama 2 70B with about a fifth of the compute; DeepSeek-V3 matches Llama 3.1 405B with about an eleventh, but needs 1.7× the memory.
Putting it to work
01If the agent is slow to show the first word, the prompt is probably too long (slow prefill); if words come out slowly, the output is probably too long or the service busy (slow decode). Cut context for the first; cap output length and ask for brevity for the second.
02The same text has different token counts in different models, so compare prices by counting tokens on your own sample text, not by the unit price alone.
03For tasks that must be stable and repeatable (extracting fields, classifying, converting formats), set temperature to 0 or close to it; raise it for divergent work such as drafting or brainstorming. Even at 0, results are not guaranteed identical (chapter 1).
04Before self-hosting an open model, estimate memory with this chapter’s formulas: weights are about parameters × 2 bytes (BF16), plus KV cache per token × context length × concurrent requests.
05When choosing an MoE model, remember that it computes fast and is cheap per token, but its memory must cover all its parameters, so it is not easy to host.
Further: eight advanced variants
Variant
How
Effect on serving
Sliding window
Local layers see only the last w tokens, and a few global layers handle long-range recall (Gemma 3 has 5 local layers per global one, w = 1,024; Mistral 7B is all local, w = 4,096)
At 128K context, Gemma 3’s KV is about a sixth of an all-global model’s
Multi-head latent attention (MLA)
Each token caches one compressed vector, from which K and V are rebuilt when needed
DeepSeek-V2 has 93% less KV cache than DeepSeek 67B and 5.8× the generation throughput
Linear attention
A fixed-size state updated as tokens are read; Gated DeltaNet decays the old state before writing (Qwen3-Next pairs 36 GDN layers with 12 attention layers, and Kimi Linear and Qwen3.5 also run 3 to 1)
Three quarters of the layers keep only a constant-size state per request
State-space models
A small state per channel whose decay depends on the token (Mamba’s selective mechanism; Mamba-2 recasts it as matrix multiplication)
No growing KV cache; Mamba reports up to 5× a transformer’s generation throughput
Sparse attention
A light indexer scores past tokens and each query reads only the top 2,048 (DeepSeek-V3.2)
Compute per token is capped, but KV for every token must still be stored
Looped transformer
The same block is reused K times
Extra depth adds compute and latency but no weights
Multi-token prediction
Predict several tokens at once
The original material only names it
Diffusion language models
Mask the positions to be generated, then fill them in over a few parallel refinement steps (LLaDA 8B, Dream 7B)
One pass fills several tokens, but fewer steps mean lower quality
Self-check
Which stage is bound by memory bandwidth and sets how many tokens per second come out?
Prefill processes the whole prompt at once, is compute-bound and sets time to first token; decode needs a full pass per token and is bandwidth-bound.
Self-check
Llama 3 70B has 80 layers, 8 KV heads and head dimension 128, in BF16. About how large is its KV cache per token?
2 × 80 × 8 × 128 × 2 B = 320 KB; if every query head had its own KV (MHA, 64 sets) it would be 2.5 MB.
10CHAPTER 10
GPU basics: why decode is bandwidth-boundGPU 基础：为什么 decode 受带宽限制
Key points
A GPU trades a few fast cores for thousands of simple ones, and hides the wait for memory by switching between thread groups (warps).
Every compute program (kernel) is bounded by three things: peak compute, memory bandwidth and memory capacity; the “ridge point” on a roofline chart separates the two kinds of bottleneck.
Decode serving one request at a time uses 0.5% of an H200’s peak compute; batching helps, but the attention part of decode stays bandwidth-bound.
Compute grows faster than bandwidth in each new GPU; raising real bandwidth means keeping more bytes on the way at once.
A CPU is a sports car, fastest for two people; a GPU is a convoy, slower per trip but carrying thousands at once. LLMs suit GPUs because every token does the same matrix multiplication on the same weights.
Warp switching hides the wait for memory. A warp is 32 threads that execute together on a GPU. While one warp waits on memory, another issues instructions, so the compute units stay busy. Each SM on A100, H200 and B200 has 4 warp schedulers managing up to 16 warps each, 64 resident warps in all.
SIMT (single instruction, multiple threads). One warp instruction drives 32 threads, paying for fetch, decode and scheduling once, which does the work of 32 ordinary instructions.
Branch divergence. When threads in a warp take different sides of an if, the two paths run one after the other; if f uses 8 of 32 lanes and g uses 24, part of the lanes sits idle in each.
What the GPU gives up
What it gains
Single-thread speed (modest clocks, no speculation, so serial code is much slower)
Chip area and power spent on thousands of arithmetic units
Parallelism must be spelled out by the programmer (about 270K threads fill 132 SMs, and small jobs leave the chip idle)
Latency hidden by switching warps rather than by caches and speculation
Shared control flow (threads in groups of 32, so divergence costs throughput)
One fetch and decode drives 32 cores
Little on-chip storage per thread (about 240 bytes at full occupancy)
Every thread’s state stays resident, so switching warps costs nothing
Three numbers and the roofline
Memory bandwidth rises each generation, but more slowly than compute
HBM bandwidth, TB/s; Rubin figures are preliminary
2 TB/sA100
4.8 TB/sH200
8 TB/sB200
22 TB/sRubin
NVIDIA datasheets, original lecture notes
A100
H200
B200
Rubin
Peak BF16 compute (how fast the math runs)
312 TFLOP/s
989 TFLOP/s
2,250 TFLOP/s
About 4,000 TFLOP/s
Memory bandwidth (how fast weights and KV cache are read)
2.0 TB/s
4.8 TB/s
8 TB/s
About 22 TB/s
Memory capacity (how much weight and KV fits)
80 GB HBM2e
141 GB HBM3e
180 GB HBM3e
288 GB HBM4
Compute is the engine, bandwidth the fuel line, capacity the tank. A kernel takes about the larger of “bytes to read ÷ bandwidth” and “operations to do ÷ peak compute”.
Decode, one request at a time: compute waits for data
3.3 ms per token
Reads 16 GB of weights from memory
Each weight is used once, so 0.016 ms of math waits on 3.3 ms of reading
Tensor cores idle 99.5% of the time
Prefill, 2,048 tokens: compute is busy
33 ms per prompt
Memory reads go quiet once the weights are in
Each weight serves 2,048 tokens, so compute sets the speed
33 TFLOP, close to peak
The roofline model finds the bottleneck from arithmetic intensity: how many operations are done per byte moved from memory (FLOPs ÷ bytes). The ridge point is peak compute ÷ bandwidth, about 206 FLOP/B on an H200. Below the ridge a kernel is bandwidth-bound (speed = bandwidth × intensity); above it, compute-bound (speed = peak).
Operation
Arithmetic intensity (b = bytes per element)
Vector add, length N
1/(3b)
Matrix times vector (GEMV, the main operation in decode), N × N, each weight used once
About 2/b
Decode attention, each KV entry used once per head
2/b
Square matrix times matrix (GEMM), each element reused about N times
2N/(3b)
Decode for one request at a time is a GEMV at about 1 FLOP/B, which on an H200 reaches only about 5 of 989 TFLOP/s, 0.5% of peak. Higher utilization needs higher arithmetic intensity: bigger batches, quantization (storing weights in fewer bits) and less KV traffic. Batching pushes the weight GEMV toward the ridge (intensity 256 at batch 256), but decode attention stays at 2/b, because every request reads its own KV.
As context grows, most bytes read per decode step come from the KV cache
Total bytes read from memory per decode step, GB; Llama 3 8B, batch 16, BF16; weights fixed at 16.1 GB
18.2 GB1K
24.7 GB4K
33.3 GB8K
50.5 GB16K
84.8 GB32K
Computed from the Llama 3 paper and the H200 datasheet, original lecture notes
Here the KV cache is 128 KiB per token and grows with batch and length; at 4.8 TB/s a step goes from 3.8 ms at 1K of context to 17.7 ms at 32K. Lower precision pushes the ridge further right: Rubin’s dense NVFP4 needs about 1,600 FLOP per byte to be compute-bound. From A100 to Rubin, BF16 compute grows about 13× and bandwidth about 9×, so kernels are increasingly bandwidth-bound.
Inside the H200
The H200 has 132 SMs sharing a 50 MB L2 cache and 141 GB of HBM3e memory (6 stacks, 4.8 TB/s), linked to other GPUs over NVLink at 900 GB/s. Each SM has 4 partitions, each with 64 KB of registers, 32 FP32 cores and a tensor core, issuing one warp instruction per cycle; the SM shares 256 KB of L1 cache and shared memory and holds 2,048 resident threads.
A tensor core is a unit built for small matrix multiply-adds: one instruction computes D = A × B + C (say 16×16 by 16×8), so one mma is 2,048 multiply-adds, 64× the work of an ordinary multiply-add instruction (FMA). Matrix multiplication is over 99% of LLM compute; softmax and normalization use the ordinary FP32 cores. Tensor cores are fast for four reasons: fewer instructions per operation; dedicated multiply-add circuits; operand reuse (one value of A serves many columns, one value of B many rows); and low precision (FP16, BF16, FP8), which raises throughput and shrinks the data, usually with results accumulated at higher precision.
Storage level (H200)
Capacity
Latency (Hopper)
Bandwidth
Registers
256 KB per SM
In the pipeline
Read directly by each instruction
Shared memory / L1
256 KB per SM
About 30 cycles (L1 hit about 40)
128 B per cycle per SM, about 33 TB/s across 132 SMs
L2 cache
50 MB
About 250 cycles
About 12 TB/s on Hopper
HBM3e memory
141 GB, 6 stacks
About 500 cycles
4.8 TB/s
Host memory and other GPUs
Microseconds
NVLink 4 at 450 GB/s each way; PCIe 5.0 about 64 GB/s
Memory is a 250-nanosecond pipe
Little’s law: bandwidth = bytes on the way ÷ latency. A simply written kernel keeps about 4 KB on the way per SM and reaches 2.2 TB/s, 45% of peak; filling 4.8 TB/s needs about 9 KB per SM (4.8 TB/s × 250 ns ÷ 132). The hardware fixes the pipe’s length, so the only way to raise bandwidth is to put more bytes in it: 16-byte vector reads, four independent reads per thread at once, and bulk asynchronous copies (TMA).
H200 (Hopper)
B200 (Blackwell)
Rubin
Memory
141 GB HBM3e
180 GB HBM3e
288 GB HBM4
Bandwidth
4.8 TB/s
8 TB/s
22 TB/s
Bytes needed on the way, whole GPU
1.2 MB
2.0 MB
5.5 MB
Bytes needed on the way, per SM
9 KB
14 KB
25 KB
New data-path features
TMA, clusters
TMEM, 2-SM MMA
Inline TMA descriptors
Further detail: coalescing merges 32 threads’ consecutive 4-byte reads into one 128 B request; 16-byte vector reads let one warp instruction move 512 B; cp.async (Ampere and later) copies into shared memory without passing through registers; Hopper’s TMA lets one thread start a whole block copy, putting tens of KB on the way from one instruction, and TMA multicast delivers one L2 read to every SM in a cluster of up to 8 thread blocks; Blackwell keeps results in 256 KB of tensor memory, freeing registers for more reads, and pairs two SMs on one matrix multiplication, each fetching half of the shared block.
From one GPU to many
HGX H200 (8-GPU node)
GB200 NVL72 (whole rack)
Makeup
8 GPUs linked by NVSwitch, 8 × 400 Gb/s network cards to the outside network
4 GPUs plus 2 CPUs per tray, 9 trays linked by NVLink switches
Memory bandwidth
4.8 TB/s
8 TB/s
NVLink
900 GB/s
1.8 TB/s
RDMA (direct memory access across machines)
50 GB/s
100 GB/s
From G80 (2006) to B300 (2025), peak compute grew about 43,000× and bandwidth only about 90×. The twenty-year arc runs from graphics to general-purpose computing (CUDA, caches, HBM2, NVLink, tensor cores) and on to low-precision AI engines (TF32, BF16, FP8, TMA, FP4, TMEM, NVL72).
Putting it to work
01When reading a GPU’s or an inference service’s specs, don’t look only at compute (TFLOPs). For LLM generation, especially when serving few requests at once, speed is set mostly by memory bandwidth, and the model size and context length you can fit are set by memory capacity.
02The longer the context, the more KV must be read for every generated token, and the slower generation gets. Stuffing a very long context into an agent costs speed at every step, not just money.
03When you host a model yourself, the main levers on speed and cost are batching requests (use an inference framework with continuous batching), quantizing the weights and shortening context. A single request with a GPU to itself leaves almost all of its compute idle.
04To estimate the best-case generation speed: every token must read all the weights at least once, so weight bytes ÷ memory bandwidth is the lower bound on time per token for one request. For example, 16 GB of weights at 4.8 TB/s takes about 3.3 ms.
05Compute grows faster than bandwidth in each new GPU, so optimizations that lower precision and cut data movement will matter more and more; when choosing models and services, check whether they support low precision and KV compression.
Further: other accelerators
Chip
Traits
Programming
AMD MI355X
8 compute chiplets that software sees as one GPU; 256 MB Infinity Cache; 64-thread groups (wavefronts); 5.0 PF dense FP8, 288 GB, 8 TB/s
ROCm and HIP; PyTorch, vLLM and SGLang run; Triton or Composable Kernel
Google TPU Ironwood
Weight-stationary systolic arrays in a 3D torus network of 9,216 chips; 4.6 PF dense FP8, 192 GB, 7.4 TB/s
JAX or PyTorch/XLA; Pallas for custom kernels
Amazon Trainium3
8 NeuronCores per chip, each with tensor, vector and scalar engines; 32 MB of software-managed SRAM per core, no hardware cache; 2.5 PF MXFP8, 144 GB, 4.9 TB/s
Neuron SDK; NKI moves data blocks explicitly from Python; no CUDA
Cerebras WSE-3
A whole wafer as one chip, 44 GB of on-chip SRAM, no off-chip memory
Larger models must stream weights in while computing
Groq LPU
230 MB SRAM, no off-chip memory
An 8-bit 70B model needs about 300 chips
Microsoft Maia 200
272 MB SRAM plus 216 GB HBM3e
Available only inside Azure
Self-check
When an H200 runs decode for one request at a time, what share of its peak compute does it use?
A one-request GEMV is about 1 FLOP/B, which at 4.8 TB/s reaches only about 5 TFLOP/s, 0.5% of 989 TFLOP/s. Utilization rises with bigger batches, quantization and less KV reading.
Self-check
What limits each operation on an H200?
Square matrix times matrix, N = 8192
A one-request GEMV
Decode attention
Prefill of 2,048 tokens
The H200 ridge point is about 206 FLOP/B: below it a kernel is bandwidth-bound, above it compute-bound.
∞SOURCES
Further reading and notes on sources
This handbook is compiled from public lecture material and the papers, technical reports and vendor documents it cites. To go deeper on a topic, start with these primary sources: ReAct (arXiv 2210.03629); Liu et al., “Lost in the Middle” (TACL 2024); Chroma’s Context Rot tests; Breunig, “How Contexts Fail” (2025); Lewis, “Same Model, Different Harness” (arXiv 2608.26218); Anthropic, “Building a C compiler”; OpenAI’s Codex report and internal usage report (June and September 2026); Meta’s Llama 3 paper; NVIDIA whitepapers and datasheets; and each benchmark’s original paper.
Charts are redrawn from the values in the source material, with the source under each chart; “original lecture notes” marks figures given there without an earlier source. The attention and FFN totals for Llama 3 70B are the per-layer figures times 80; every other value is as given in the sources.
The “Putting it to work” section in each chapter is practical advice written from that chapter’s content and adds no data beyond the source material.
Product names appear only for specific case studies, data sources or tool lists; general agent mechanisms are described in general terms, so check your own agent’s documentation for exact commands and file names.