Agents and AI system optimizationCS2680A study handbook for CS2680, lectures 1 to 10
The course treats an agent as a systems problem. The model sets the ceiling, the harness sets realized performance, and the serving system and GPU set cost and speed. Lectures 1 to 8 cover using and designing agents: the loop, tools, context, memory, multi-agent systems, reliability, evaluation and cost. Lectures 9 and 10 cover how LLM inference and GPUs work. Each lecture opens with its key points and ends with a self-check.
SCROLL
01LECTURE 1
Course map: AI systems and agentic systems课程地图：AI 系统与智能体系统
Key points
AI systems ask how to train and serve models at scale, for performance, efficiency, reliability and sustainability; here that means LLM systems.
An agent uses a model to reason, call tools and act over many steps until a task is done. It is a new kind of computer: the LLM is the CPU, context is RAM, tools are peripherals, the harness is the OS.
Agent serving is a long chain of inference and tool calls, and each property of the workload names a systems problem.
The objective is not throughput but task success under latency, cost, resource, reliability and security constraints.
Five ways AI systems differ from traditional systems
Dimension
What differs
Compute
Heterogeneous: the CPU tokenizes, launches kernels, moves data and runs tool calls, the GPU does matmuls. Prefill is compute-bound, decode is memory-bandwidth-bound. Execution time depends on output length, which complicates scheduling
Non-determinism
Floating-point addition is not associative, and a shift in the seventh decimal can flip the next token. Sources are the runtime (warp scheduling, batch size), software (fused multiply-add vs two ops, autotuning) and hardware (CUDA vs tensor cores, FP8 vs micro-scaling formats). Approximate computing is acceptable
Memory, storage, communication
Weights run from GBs to TBs, the KV cache takes tens to hundreds of KB per token, and a prefix cache reuses KV across requests. Data movement is the bottleneck. Multi-GPU serving needs pipeline, tensor, expert and context parallelism over NCCL, NVLink, InfiniBand and RDMA
Fast-moving models and workloads
Attention moved from MHA to GQA, MLA, NSA and linear attention. Workloads moved from multi-turn chat, long reasoning and RAG to agents, which need long context, prefix caching, routing and load balancing
Reliability and power
9% annual failure rate: a 16K-GPU cluster loses a GPU about every 3 hours. A GPU draws 700 to 1000 W, and server power rose from 800 W to 6,000 W
Server power rose about eightfold with GPUs
Power per server, W
CPU server800 W
GPU server6,000 W
Lecture 1 slides
What an agent is
An AI agent is an autonomous software system that uses an AI model to reason about a goal, use external tools and execute multi-step actions until a task is finished. Its work breaks into five steps:
01Plan: break the task into sub-steps
→
02Act: call external tools
→
03Observe: interpret the results
→
04Adapt: revise the plan on new information
→
05Deliver: return the final result
Agents are needed because LLMs are uneven. They are superhuman at recalling facts, language, pattern matching, generating code and working across domains, and weak at persistent memory, reliably understanding context, avoiding hallucination, knowing what they don’t know and resisting adversarial input. Tools and memory fill the gaps.
An agent can be read as a new kind of computer: the LLM is the CPU, the context window is RAM, tools are peripherals, the agent harness is the operating system and the agent is the application. Its programming model and failure model are entirely different.
Why agents are a systems problem
Traditional LLM serving is one request, one inference, one response. Agent serving is a chain: infer, tool, infer, tool, retrieval, infer. The critical path spans GPU, CPU, storage, network and external services, which brings dependencies between steps, highly variable step latency, blocking on tools, rapidly growing state, concurrency and speculation, and failures anywhere in the trajectory.
Property of the agent workload
Systems problem it names
Long trajectories
State and context management
Huge context
Memory hierarchy and retrieval
Repeated prefixes
Prefix caching
Branching reasoning
Speculative execution
Tool dependencies
Dependency-aware scheduling
Different models
Routing
Variable tool latency
Asynchronous execution
External side effects
Security and isolation
Multiple agents
Orchestration and communication
Long-running tasks
Checkpointing and fault tolerance
The stack, top to bottom: application and user; agent and harness (planning, context, memory, tools, subagents), beside the external environment and tools (web, files, APIs, humans); the model; the serving system (batching, scheduling, routing, caching); runtime and kernels (CUDA, Triton, collectives); hardware (GPU, CPU, memory, network, storage).
What to optimize
Throughput is the wrong objective for an agent. The objective is to maximize task success subject to latency, cost, resources, reliability and security.
What we actually optimize
Task success and accuracy
End-to-end latency
Tokens per task
Dollars per successful task
Energy per successful task
Useful, but insufficient
TTFT and TPOT
Tool-call latency
GPU utilization
More tokens per second can still mean a slower, costlier agent. Observability is a first-class problem: for agents the distributed trace becomes a nested trajectory of sessions, turns, agent steps, model requests, tool calls and subagents. You cannot optimize what you cannot observe.
Questions the course answers
Batching and latencyWhy does batching raise throughput but hurt agent latency? When should an agent use a larger model?
Where state livesWhere should KV and prefix state live? How should requests from one agent be scheduled?
Overlap and delegationCan tool calls overlap with inference? When should we spawn a subagent?
Routing and recoveryHow do we route across heterogeneous GPUs and models? How do we recover a 30-minute agent after a failure?
The objectiveHow do we optimize dollars per successful task rather than tokens per second?
The course does not cover training (inference dominates compute demand), model architecture design, communication, power and energy, or new hardware design.
Also: six levels of AI coding, and agents vs assistants vs bots
Level
Meaning
0
No AI, hand-written code
1
Assist: AI autocompletion
2
Supervised: AI writes functions, you review
3
Scoped: hand off a task, review the code
4
Goal-driven: AI writes all, tests verify
5
Parallel agents: subagents split the work
Agent
Assistant
Bot
Purpose
Performs tasks autonomously and proactively
Assists users with tasks
Automates simple tasks or conversations
Capabilities
Complex multi-step actions; learns and decides independently
Responds to prompts and can recommend; the user decides
Follows predefined rules; limited learning
Interaction
Proactive, goal-oriented
Reactive, responds to requests
Reactive, responds to triggers
Self-check
Which does the course say we should actually optimize for an agent?
The slides call throughput the wrong objective; TTFT and GPU utilization are useful but insufficient. The goal is task success under constraints, measured as cost and latency per successful task.
Self-check
Which systems problem does each workload property name?
Repeated prefixes
Variable tool latency
External side effects
Long-running tasks
02LECTURE 2
How Claude Code works: the loop, memory and extensionsClaude Code 如何工作：循环、记忆与扩展
Key points
One prompt expands into many model calls and tool runs: the model proposes, the harness checks permission and executes, the result goes back into context.
Context is rented working memory; CLAUDE.md and auto memory are files loaded every session, and stale memory is worse than none.
Each extension does one job: a skill says how, MCP reaches outside, a hook enforces, a plugin packages and distributes.
01Gather context: prompt, files, tool results
→
02Model proposes an action, e.g. Bash(“pytest”)
→
03Harness checks permission and runs it
→
04Result goes back into context
Facts about the loop
Context is over 10K tokens before you type. System instructions, project environment, memory (CLAUDE.md), tools (Read, Edit, Bash, Glob, Grep, Web, Agent), and skill and MCP names and descriptions come first, then the user’s words. Skills and MCP start as names only; the full text loads when Claude reaches for them.
It does not send your repository. The repo stays on disk and Claude pulls slices on demand: Grep(“authentication”) brings back a handful of lines.
The model proposes, the harness executes. The model emits a tool request; the harness resolves it, checks permissions and runs it on your machine; the tool_result goes back into the model’s context. Every request is conditioned on the previous inference and the previous tool result.
Permissions sit between intent and execution. A tool request is a proposal that needs your approval or another model’s (table below).
Context grows, then gets compacted. A typical window is 1M tokens (256K was common earlier in the year). Prefix-cache reads account for most of the cost. Compaction summarizes the old trajectory and re-injects CLAUDE.md and memory.
Permission mode
Meaning
Manual
You confirm each request
Accept edits
Edits pass automatically; other requests still ask
Bypass permissions
Everything runs directly
Auto
A small model checks each request
A request ends in one of three ways: allowed (the command runs), ask (you confirm first) or denied (nothing happens).
Memory: context is rented, memory is owned
Agents are stateless between runs; every session starts from zero, and memory is just files loaded at session start. Context is working memory and dies with the session; persistent memory survives and is reloaded. Memory answers four problems: repeated instructions, repeated discovery, decisions whose reasons vanish at compaction, and sessions that make different choices. Memory is how instruction becomes convention.
CLAUDE.md
Auto memory
Who writes it
You
Claude
Contains
Instructions and rules
Learnings and patterns
Scope
Project, user or organization
Per repository, shared across worktrees
Loaded
Every session
Every session (first 200 lines or 25KB)
Use for
Standards, workflows, architecture
Preferences, corrections, project context
CLAUDE.md files load broadest scope first, and the closest file is read last: managed policy (org-wide, cannot be excluded), user ~/.claude/CLAUDE.md, project ./CLAUDE.md (shared through source control), local ./CLAUDE.local.md (personal, gitignored).
Belongs in CLAUDE.md
Commands: how to run tests, the build, the linter
Hard constraints: never commit to main, never edit generated files
Facts the agent cannot cheaply infer: layout, ownership, gotchas
Does not belong
What the model already knows about the language or framework
Style a linter or formatter already enforces
Long prose, changelogs, anything secret
Keep each file under 200 lines, in Markdown headers and bullets, with concrete and consistent rules. When it grows, split it into .claude/rules/, one topic per file; with a paths: frontmatter a rule loads only when Claude touches matching files.
Auto memory lives in ~/.claude/projects/<project>/memory/, each file tagged with a type: user (role and preferences), feedback (corrections and confirmed approaches), project (ongoing work and decisions not in the code), reference (information outside the project). MEMORY.md is the index, and its first 200 lines load each session; topic files are read only when relevant. Commands: /memory (browse, edit, disable), /context (which instruction files loaded), /init (a starter CLAUDE.md), /doctor (content that may not belong in CLAUDE.md).
Remembering has a cost. Every permanent token is rent paid at the start of every session, and memory competes with the task for the same window. Stale memory is worse than none because it actively misleads. Treat memory like code: review it, prune it, keep it in version control.
Tools and four extension mechanisms
Core tool
What it does
Read
Opens a file or inspects state; the most used tool
Search
grep and glob: find it without reading everything
Fetch
Pulls a page or doc, bringing untrusted text inside
Edit / Write
Changes files; the first tool with consequences
Shell
Build, test, git; the whole machine unless sandboxed
Delegate
Spawns a subagent that returns a summary
The model decides which to use.
Mechanism
Problem it solves
Where it attaches in the loop
Skill
How the agent should perform a task (knowledge and procedure)
Model reasoning
MCP
Which external data and actions it can reach
Tool selection
Hook
What happens automatically at a lifecycle event
Any stage: inspect, block or react
Plugin
How extensions are packaged and distributed
Bundles the other three into one installable unit
A release workflow (prepare, validate, publish to GitHub, notify the team) shows the split:
01Skill: follow the release procedure
→
02MCP: create the GitHub release
→
03Hook: block publishing when tests fail
→
04Plugin: share the complete setup
A skill is a folder: SKILL.md (the name and description control discovery; the body holds the procedure) plus references/ and scripts/. Descriptions load at session start; the body loads only when the skill runs. A skill is invoked by the model when its description matches the task, or by the user as /<skill-name>. Side-effecting workflows should usually require explicit invocation, and clear, distinct descriptions reduce wrong selections. Think of it as context engineering.
MCP (Model Context Protocol) connects an agent to external services through a standard protocol: GitHub, databases, monitoring, Figma, Slack, a browser. A call takes six steps: Claude identifies a needed action, the MCP client calls a tool, the MCP server handles the connection, the external service performs the operation, structured results return, and Claude evaluates them. It is almost identical to a local tool, but runs in another process.
Hooks fire on events; model reasoning is not an event. A hook can run a command, an HTTP request, an MCP tool, a prompt or a subagent.
01Session start: load context, log
→
02User prompt: validate or add
→
03Before tool use: inspect or block
→
04After tool use: format, test, log
→
05Before stopping: verify tests
A plugin packages skills, hooks, agents, .mcp.json and .lsp.json, and gives one-command installation, pinned versions, namespacing, team reuse and marketplace distribution.
Inside .claude/: CLAUDE.md and rules/ are memory; settings.json holds permissions, hooks and environment variables; skills/ and commands/ are workflows; agents/ and hooks/ are delegation and guardrails. The same layout at ~/.claude/ applies to user scope, and project settings win over user settings.
Requirement
Mechanism
Apply the company API conventions
Skill
Read an issue from Jira
MCP
Run formatting after every edit
Hook
Guide a multi-step deployment
Skill
Send a completion notification
Hook plus MCP
Block writes to production
Hook or policy
Share everything across repositories
Plugin
Further: MCP in detail
Before MCP, N AI clients and M services needed N×M integrations. MCP is an open, stateless protocol: the host (such as Claude Code) runs one MCP client per server, and each MCP server exposes a system’s capabilities as tools.
MCP primitive
What it is
Example
tools
Actions Claude can call
create_issue(…)
resources
Readable data, such as an issue or a schema
github://issues/123
prompts
Reusable templates a server exposes, user-controlled
Shown in Claude Code as /mcp__github__triage_issue
They cooperate: to triage issue #123, the prompt supplies the organization’s triage procedure, the resource supplies the issue, and a tool retrieves current diagnostics.
Security: least privilege, scoping the token to the repo, table or channel you need; separate reads from writes and keep mutations behind approval; never treat returned content as instructions; vet third-party servers, which can carry prompt injection. Before connecting, ask what the server can read and what it can change.
An MCP gateway is a control layer between an AI client and several MCP servers that aggregates, routes, enforces policy, brokers identity, observes and scales. It is worth it when many users, agents, servers, credentials or compliance rules must be governed consistently, but it is a trust boundary with risks of broad credentials, lost end-user identity, name collisions, secrets in logs and a single point of failure. The 2026-07-28 spec is the largest revision since launch: a stateless core in which any request can land on any instance; Mcp-Method headers that let gateways route without reading the body, with list results cacheable by TTL; and Tasks for long-running work and MCP Apps for server-rendered UIs, shipped as extensions.
Self-check
The design exercise from the slides: which mechanism fits each requirement?
Apply the company API conventions
Read an issue from Jira
Run formatting after every edit
Guide a multi-step deployment
Block writes to production
Share everything across repositories
A skill carries knowledge and procedure, MCP external access, a hook event automation, a plugin packaging and distribution. Blocking production writes can also be a policy.
Self-check
Which belongs in CLAUDE.md?
CLAUDE.md holds commands, hard constraints and facts the agent cannot cheaply infer; known knowledge, linter-enforced style, long prose and changelogs do not belong.
03LECTURE 3
Multi-agent use and Programming 3.0多智能体用法与 Programming 3.0
Key points
Changing only the instructions changes the workflow: Superpowers sets a development process with skills and a session hook; Caveman compresses tokens both ways with skills, hooks, MCP and a local proxy.
Scaling has three axes: parallel agents buy wall-clock time, subagents buy focus, long-running agents buy scope; agent teams pay off only when the work needs discussion.
Programming 3.0 is about uncertainty: do not expect the agent to be deterministic; make the surrounding system dependable and match autonomy to risk.
Field data: delegated tasks grew much larger in six months, but the longer the task, the fewer succeed without a human stepping in.
Case study: Superpowers
Superpowers is a software-development plugin. A default agent starts coding on a feature request and fixes by iteration; with the plugin, the work follows a fixed path:
01Human types the request
→
02Clarify the problem
→
03Approve a design
→
04Write a detailed plan
→
05Test-driven build
→
06Review and verify
→
07Finish or integrate the branch
It consists of nine skills:
Skill
Responsibility
using-superpowers
Rules for discovering and invoking skills
brainstorming
Turns a request into an approved design
writing-plans
Decomposes the design into concrete tasks
using-git-worktrees
Creates an isolated workspace
test-driven-development
Enforces red, green, refactor
systematic-debugging
Finds root causes before proposing fixes
requesting-code-review
Reviews implementation against the plan
verification-before-completion
Requires evidence before reporting success
finishing-a-development-branch
Merge, pull request, retention or cleanup
A skill file has six parts: name, description, instructions, checkpoints, references and scripts. A process skill can also define entry conditions, required and prohibited actions, human approval gates, verification requirements and the next skill.
Installing skills makes them available but does not make the agent check them. Superpowers solves this with a SessionStart hook that runs at startup, after clear and after compaction: a script reads the bootstrap skill, escapes it as JSON and returns it as additionalContext.
Design question
Superpowers’ answer
How is complex behavior represented?
Small, composable skills
How does the agent remember to use them?
A bootstrap skill injected by a session hook
How is context kept manageable?
Task skills load only when relevant
How do stages coordinate?
Each skill names its gates and successor
How do users install it?
Everything packaged as a plugin
Does every plugin need commands or MCP?
No; add them only when needed
Case study: Caveman
Caveman compresses an agent’s tokens in both directions, for three reasons: logs, diffs and tool output crowd out the work; every turn is paid for and waited on; and trimming late at compaction drops what mattered. For example, 786 lines of test output become “3 tests fail: ETIMEDOUT at client.ts:88”.
65%Reported reduction in output tokens; the skill adds input tokens, so whole-session savings are lower
2Directions: the skill changes how the model writes, the proxy changes what it reads
5MCP tools: compress, byte-exact retrieve, stats, JSON to TOON and back
It uses all five extension mechanisms plus a local proxy. A skill sets the concise style (keep technical terms, commands, paths, code, errors, numbers, units and negation; drop filler, repetition, pleasantries and hedging). Commands let the user pick a mode (/caveman ultra, /caveman off and others). A SessionStart hook decides and stores the session’s mode and restores the rules after compaction; a UserPromptSubmit hook detects mode changes mid-conversation. MCP exposes compression and retrieval of the original. The plugin packages it all. The proxy compresses by type: errors, stack traces, head and tail for logs; imports, signatures and types for code; file and hunk headers and changed lines for diffs; keys and structure for JSON.
Hooks can intercept requests but cannot transparently rewrite the request to the LLM, so a proxy is needed. The proxy intercepts every request and changes the context; MCP runs only when called and cannot. If you write your own harness, you can do it directly.
Three ways to scale
Parallel agents
Subagents
Long-running agent
How
Independent tasks, each in its own worktree
Delegate a bounded task to a fresh context and get back a summary
Carry a larger job across many steps
Buys
Throughput (wall-clock time), not answer quality
Throughput and focus: the parent never sees the intermediate mess
Scope: more work with less intervention
Parallel agents by hand.claude -w feature-payments and claude -w bugfix-auth each get a git worktree: a separate working directory and branch that share the repository’s history, so edits never touch another session’s files, with no stashing and no second clone. With three sessions you can hold the plan. With twenty you need a clone of yourself: the plan lives in your head, sessions cannot see each other, you reconcile the branches, and nothing is written down to rerun.
Subagents are specialized assistants with their own context window, tools and permissions, used to preserve context, enforce constraints, reuse prompts across projects, run in parallel and control cost (for instance with Haiku). A custom subagent is a Markdown file with YAML frontmatter (name, description, tools, model) whose body becomes the system prompt, kept in .claude/agents/ (project) or ~/.claude/agents/ (user). Subagents nest up to five levels, and two to three is the sweet spot; Agent must be in tools before a subagent can spawn children.
Built-in subagent
Use
Explore
Read-only search of the codebase; write and edit denied; thoroughness set per call
Plan
Research during plan mode, so exploration stays out of the main window
General-purpose
Complex multi-step work that needs both exploration and action
Explore and Plan skip CLAUDE.md and git status to stay fast and cheap. Ways to invoke, in increasing control:
01Name it in the prompt: Claude decides whether to delegate, for one request
02@agent-name: guaranteed to run for one task
03claude --agent name: the whole session adopts its prompt, tools and model
Keep in the main conversation
Delegate to a subagent
Iterative back-and-forth
Verbose output you won’t reuse
Phases that share context
Tool or permission restrictions
Quick, targeted changes
Self-contained work returning a summary
When latency matters
Parallel, independent work; simple work on a small model
Agent teams (experimental, off by default) are several Claude Code sessions with one lead and several teammates. The lead spawns teammates, coordinates and synthesizes; each teammate is a separate instance with its own context; a shared task list holds work items to claim; each agent has a JSON mailbox, and teammates talk to each other directly. Teammates do not inherit your conversation but load CLAUDE.md and skills automatically. Teams suit review from several angles, independent modules, debugging with competing hypotheses and cross-layer work; for sequential tasks, same-file edits or heavy dependencies, use one session or subagents.
Subagents
Agent teams
Communication
Back to the caller only
Teammate to teammate
Coordination
The lead manages the work
Shared task list
Best for
Result-only tasks
Work that needs discussion
Token cost
Lower
Higher
A dynamic workflow is a JavaScript script that orchestrates many subagents in the background, for a codebase-wide bug sweep, a 500-file migration, cross-checked research or a plan from several angles. Type ultracode in the prompt, or use /effort ultracode and let Claude decide.
Long-running agents./goal “npm run lint exits with zero errors” hands over a verifiable end state; an evaluator rechecks it after each attempt, and the run stops when the goal holds or --max-tries runs out. “All integration tests pass” works; “refactor the code” does not. /loop 5m “check CI, fix whatever broke” re-runs a prompt on an interval until you press Esc; a default prompt can live in .claude/loop.md.
Case study: sixteen agents built a C compiler
16Agents in parallel, about 2,000 sessions over two weeks
~100KLines of Rust that build Linux 6.9 on x86, ARM and RISC-V
2B / 140MInput / output tokens; just under $20,000 in API cost
~99%Pass rate on major test suites
Plain infrastructure did the orchestration: each agent had its own container and git checkout; tasks were claimed with lock files in current_tasks/; an agent implemented and tested, pulled everyone else’s work, then merged, pushed and unlocked; fresh sessions read progress docs to regain context; specialist agents handled docs, performance, quality and deduplication; automated tests gave a progress signal. There was no chat and no manager agent. The limits: merge conflicts were frequent; parallelism stopped helping once every agent hit the same kernel failure; part of 16-bit x86 still leaned on GCC; and the generated code was less efficient than GCC’s.
Programming 3.0 and uncertainty
The lecturer’s framing
The human writes
Through
Into
Programming 1.0
Machine instructions or assembly
Hardware
Programming 2.0
A programming language
A compiler
Machine instructions
Programming 3.0
A goal in natural language
An agent
A program, then machine instructions
Each step raises productivity and uncertainty. Karpathy’s Software 3.0 is a different framing: the 1.0 program is source code, 2.0 is neural-network weights (collect data, optimize), 3.0 is LLM prompts (natural language). Knowing where to draw the boundary between 1.0, 2.0 and 3.0 is an important engineering skill.
Uncertainty is not new. Hardware (32 vs 64 bit, endianness), environment (concurrency, unreliable networks) and software (compiler versions, dependency drift) were handled in 2.0 by fixed-width types, memory consistency models, idempotent APIs and testing before upgrades. 3.0 adds non-deterministic kernels and floating-point variance, code generation that touches the environment as it writes, sampling, and ambiguous input from a single prompt. Uncertainty sits in five layers: natural language (semantics), retrieval and context (retrieval drift, index updates, truncation), model generation (sampling, software and hardware non-determinism), the agent and tool loop (tool failure, environment change), and the generated program (usually deterministic once generated, unless it calls models or tools at runtime).
01Define: state assumptions up front and have the model confirm anything ambiguous before it executes; specify outcomes, not prompts, with requirements, invariants, constraints and acceptance tests
02Control: separate proposal from execution and validate plans before anything commits; manage context explicitly, owning memory, provenance and retrieval and expiring stale information
03Verify and govern: tighten the generate-verify loop, since cheap, frequent checks beat one expensive review at the end; measure one level deeper with tests, several metrics and synthetic workloads you can reason about
Do not expect the agent to be deterministic; make the surrounding system dependable. Match autonomy to risk.
Evidence from the field: Codex and OpenAI
Tasks delegated to Codex grew much larger in six months
Share of users tackling tasks of each size, %, December 2025 vs May 2026
≥ 1 human-hour · 2025.1235.4%
≥ 1 human-hour · 2026.0570.2%
≥ 8 human-hours · 2025.122.1%
≥ 8 human-hours · 2026.0525.6%
Codex report (June 2026), cited in lecture 3
Users tackling full-day tasks rose twelvefold. Active users grew more than five times in the first half of 2026. In the last 28 days, under 1% of individuals and 17.3% of organizations used Codex, against nearly all OpenAI workers.
28.6%OpenAI users who ran 5 or more agents at once in a week
~2.5Agent-hours per day for the median employee
~71Agent-hours per day at the 99th percentile, up 88% since April
3.14×Agent workdays over researcher workdays, September 2026
Before June 2026, agent runtime trailed human labor; now it exceeds it. Most researchers run four or more concurrent workflows, and lines changed per engineer approach eight times the pre-2025 average. The human role shifts to planner, delegator, monitor, reviewer and integrator.
The longer the task, the fewer succeed without human intervention
Tasks that succeeded with zero interventions, %, by estimated time for a human
86%<15m
76%15–30m
69%30m–1h
61%1–2h
57%2–4h
43%4–8h
40%8–16h
23%16–32h
13%32–64h
16%64–128h
OpenAI report (September 2026), cited in lecture 3
The delegation ladder, bottom to top: chatbot (AI answers, the human does the work), assistant (human and AI work together), agent (the human delegates a whole unit of work), multi-agent worker (the human coordinates several agents), systematized workflows (skills and agents run processes). The slides place OpenAI’s workforce near the bottom.
Also: CLI coding agents, mid-2026
Group
Examples
Model labs
Claude Code, Codex CLI, Antigravity CLI, Grok Build, Mistral Vibe, Kimi Code, Qwen Code
Everything included: oh-my-pi, Claude Code. Four tools only: Pi, mini-SWE-agent (about 100 lines of Python)
Self-check
Keep it in the main conversation, or delegate to a subagent?
Iterative back-and-forth
Verbose output you won’t reuse
Quick, targeted changes
Tool or permission restrictions
Latency matters
Parallel, independent work
A subagent starts fresh, sees no history and pays a latency cost to gather context; it suits self-contained work that returns a summary.
Self-check
Which works as a /goal?
/goal needs a verifiable end state that an evaluator can recheck after each attempt.
04LECTURE 4
Agent design I: the loop, tools and context智能体设计 I：循环、工具与上下文
Key points
Agent = model × harness; changing only the harness raised tasks solved on two SWE-bench variants by 1.7 to 2.3 times.
Tool design is context design: few, orthogonal tools, token-cheap output, errors that teach.
Context costs money, latency and attention, and accuracy is lowest when the answer sits in the middle.
When the window gets too large, use admission, offloading and transformation first; LLM compaction is the last resort.
ReAct and the agent loop
ReAct puts reasoning and acting in one loop: a thought (what do I know, what is missing, what next), an action (a structured call the environment can run) and an observation (the environment answers with facts). Reasoning alone is never checked, so small errors compound into confident hallucination; acting alone has no plan behind it, so the model repeats itself. In ReAct, thoughts steer the next action and observations correct the next thought, until the observations answer the question or a stopping rule ends the run.
01Start the message list with the system prompt and the user request
02Loop up to a step limit: call the model and append its reply
03If it gives a final answer, return it
04Otherwise, for each tool call: validate the arguments, run the tool, append the serialized result as a tool message
05Raise an error when the step limit is exceeded
As a harness designer you write the tool definitions, assemble the context, validate requests, check permissions and run them, and optionally process results and manage context. A coding agent’s three components share one context window and are designed together: the model plus loop; tools and guardrails (permissions, sandboxing, approval); and state, meaning what is in the window now plus what the agent carries between steps.
Tool design: search and read
Approach
Finds code by
Best when
Main cost
glob / grep
Literal names and paths
The name is known or guessable
Turns and context per search; lexical matching misses paraphrase
Semantic index
Meaning of embedded chunks, or an Aider-style repo map (file tree plus ranked symbols)
The wording won’t match the code
Building and re-syncing the index
Extrinsic signals
Stack traces, failing tests, git history
A report already points at code
Exists only for some tasks
Language server (LSP)
The compiler’s symbol graph: definitions, references, types
Edits must follow references, as in a rename
A live server per language; slow cold start
In practice, start with grep and reach for an index or LSP when names run out.
Tool design: editing
Approach
How
Risk
Whole-file rewrite
The model regenerates the file
Output scales with file size; untouched code drifts on long files
Unified diff / patch
The model emits a patch that a tool applies
Hunks fail when line numbers or context go stale
Search and replace
An exact anchor string plus its replacement
Fails on ambiguous, stale or whitespace-off anchors
Structured / AST edit
An operation on a symbol, such as rename or add parameter
Per-language tooling and a limited set of edits, but valid by construction
Tool design: execution and other tools
The shell is the most powerful and most dangerous tool. One call can chain, filter, loop and retry; it reaches the filesystem, network, package managers and binaries; files and processes outlive the call; whatever the shell can do, the agent can do. Design for isolation (where code runs and what it can reach), failure signals the agent can act on, output that is enough but not too much, duration (long-running and background processes) and budgets (when to stop retrying and hand back). Five other common tools: memory and notes (plans survive compaction), subagents (exploration returns a shortlist rather than 40 file reads), web search and fetch (read the installed source first, then the web), browser and visual feedback (screenshot the render and feed it back), and asking the user (one cheap question beats a confident wrong patch).
Few, orthogonal toolsOverlapping tools make the model dither between them
Purpose-built, not genericsearch_orders(customer_id) beats a raw run_sql
Token-cheap outputTruncate, paginate, summarize: tool design is context design
Errors that teachReturn the fix hint, not just “invalid argument”
Idempotent by defaultSafe to retry whenever possible
A purpose-built agent interface solves more issues than a bare shell
SWE-agent ablation on GPT-4, resolve rate, %
Purpose-built SWE-agent18%
Full interaction history15%
Full-file view12.7%
Shell only11%
No specialized editor10.3%
Lecture 4 slides (SWE-agent ablation)
From prompt engineering to goal engineering
Period
Name
Question
Concerns
2023–2024
Prompt engineering
How should we instruct the model?
Wording, examples, output formats
2025–2026
Context engineering
What should the model see?
Retrieval, memory, history, compaction
Now
Harness engineering
How should the agent operate?
Loop, tools, sandbox, retries, stopping
Next
Goal engineering
What counts as success?
Objectives, criteria, constraints, trade-offs
Technique
How
Use it when
Move on when
Zero-shot
Instruction only
The task is common and the format fits in words
The rule is tacit: few-shot
Few-shot
Instruction plus a few examples
The output shape is unusual or the rule is easier shown
The task needs steps: chain-of-thought
Role and persona
Give the model a role
Tone or judgment matters; the audience is specific
The task needs steps: chain-of-thought
Chain-of-thought
Reasoning steps first, answer last
The task has steps and errors come from skipped work
Several paths exist: tree-of-thought
Tree-of-thought
Branch, score, prune
Several routes are plausible and cheap to score
One right answer: self-consistency
Self-consistency
Sample several times and vote
One right answer and an expensive wrong call; route to a human below the floor (e.g. 3/5)
Separable stages: chaining
Prompt chaining
One job per call, validated handoffs
Work splits into stages you can validate; retry only the failed stage
The prompt itself is being tuned: meta prompting
Meta prompting
The model rewrites its own prompt
You have an eval set to score on
Gains stall: revisit the choice
Four practices: start simple and add context only when needed; partition into sections (context, instructions, tools, output spec, in the order instruction, delimiter, data); be specific and concise, since clever phrasing buys nothing; split big tasks rather than front-loading complexity. Prompting advice is model-specific; re-tune when you switch models. Some techniques have already expired.
Context engineering: context is a finite resource
Karpathy’s definition: context engineering is the art and science of filling the context window with just the right information for the next step. Context costs in three ways: money (every step re-sends the window), latency (bigger prompts prefill and decode more slowly) and attention (recall fades as the window fills, and the middle gets missed).
Accuracy is lowest when the answer sits in the middle
Answer accuracy by position of the answer among 20 documents, %
76%1st
61%5th
55%10th
59%15th
71%20th
Liu et al., “Lost in the Middle”, TACL 12 (2024), cited in lecture 4
Length costs too: in Chroma’s Context Rot test, as input grew from 1K to 400K tokens, accuracy fell to 73% on simple lookup and 46% with distractors.
An illustrative budget for one agent turn: system prompt about 2K, tool definitions about 6K, project environment about 8K, retrieved context about 25K, message history about 80K and growing. When the window gets too large there are four strategies, from preventive and lossless to lossy and irreversible:
The first three strategies shrink context by 75% to 98% without summarizing
Tokens before and after, examples from the slides
Admission · raw DOM48,000
Admission · extract only900
Offloading · 60 tool schemas42,000
Offloading · index plus 33,100
Transformation · every result96,000
Transformation · newest state24,000
Lecture 4 slide examples
Strategy
How
Admission
Guard what enters: targeted retrieval (fetch fields, not documents), shape tool output, truncate only when shaping cannot, hand the raw payload to a subagent
Offloading
Do not load until needed: progressive tool definitions, tool search (keep an index, fetch schemas on demand), tool output returned as a handle, task list kept on disk, persistent memory with a small index
Transformation
Drop history selectively: deduplication (newest state of each object), observation masking (strip tool results older than N steps), sliding window (keep the head and recent turns)
Compaction
The last resort, using an LLM: rolling summary, full summary at the limit, auto memory that persists key facts outside the window. Reserve headroom for output, preserve the system prefix, re-inject critical facts afterwards
Five ways context fails, and naming the failure makes the fix obvious: poisoning (a hallucination enters the window and is later cited as fact), distraction (history grows until the agent imitates its own past actions), confusion (irrelevant tools and documents steer the answer off course), clash (new and old information contradict and both stay) and drift (the real goal is buried and swapped for a nearby subgoal).
Changing only the harness: SWE-bench Verified rose from 43/169 to 72/169 and SWE-bench Pro from 31/316 to 72/316. The changes were compacting old tool outputs, keeping recent observations visible, detecting repetition and stalls, and intervening when stuck. Some apparent model failures are context and control-loop failures.
Further: beyond ReAct
ReAct loops every step. Reflexion loops after a failure (attempt, evaluate, critique, retry). Plan-and-execute loops on deviation (plan, execute steps, re-plan). Tree search does not loop: one wide pass explores branches in parallel and keeps the best.
Self-check
Which context strategy is each technique?
Fetch fields, not whole documents
Keep a tool index; load schemas on demand
Keep only the newest state of each object
Summarize history with an LLM at the limit
Hand the raw payload to a subagent
Strip tool results older than N steps
From admission to compaction, the strategies run from preventive and lossless to lossy and irreversible.
Self-check
“History grows until the agent imitates its own past actions.” Which context failure is this?
Poisoning is a hallucination later cited as fact, confusion is irrelevant tools and documents steering the answer, drift is the real goal swapped for a nearby subgoal.
Multiple agents are not a default; the reasons to split are specialization, isolation, parallelism and context management.
Every architecture answers three questions: who coordinates, how work is decomposed, how agents communicate. Real systems are hybrids.
A subtask states its objective, boundaries, output and completion test; parallelism helps only when the work saved exceeds the coordination cost.
Communication is a distributed systems problem: separate control from content, and handle timeouts with idempotency keys and rollback paths.
Three graphs and eight architectures
Every architecture answers three questions. The control graph: who coordinates, whether one supervisor, a hierarchy or nobody, and does that change at runtime (handoff)? The task graph: how is work decomposed, whether up front by a planner, by bidding or emergently from local rules? The communication graph: how do agents talk, by direct message, broadcast or a shared workspace?
Category
Architecture
Core idea
Best for
Weakness
Centralized
Supervisor–worker
One planner, many workers
Separable, parallel subtasks
The supervisor bottlenecks; its context grows fastest
Centralized
Hierarchical
Supervisors of supervisors
Tasks that already decompose recursively
Errors amplify; how much context crosses each boundary
Centralized
Pipeline / DAG
Each stage feeds the next through a typed contract
Clear ordered stages
Errors propagate with no path back; cheapest to debug
Decentralized
Market / auction
Agents bid on capability, cost and latency
Dynamic allocation to the cheapest capable agent
Bids are self-reported and can overstate ability
Decentralized
Blackboard
Coordination through shared state (database, filesystem, git repo)
One shared artifact
Stale state; consistency, concurrency, provenance, versioning, garbage collection
Decentralized
Committee / debate
Answer, then a judge or vote reconciles
Judgment calls
Groupthink: same model and context, same blind spot
Decentralized
Swarm
Direct negotiation, no coordinator
Local negotiation
O(n²) messages, hard conflict resolution; rare in products
Hybrid
Dynamic topology
Agents added, removed or rewired at runtime
Unknown structure; simple tickets stay cheap
Every run takes a different shape, so failures are hard to reproduce
Real systems are hybrids. A planner dispatches a dynamic task DAG to coder, researcher, tester and reviewer agents that share a git repository, re-planning when needed. That combines supervisor, hierarchy, shared memory and pipeline.
01Planner agent
→
02Dynamic task DAG
→
03Coder · researcher · tester · reviewer
→
04Shared git repository
→
05Re-plan if needed
Task decomposition
Work can be split by data (each agent takes some documents), expertise (security, performance, correctness, legal), stage (plan, implement, test, review), hypothesis (agents investigate competing explanations), tool (web, database, code execution) or authority (different administrative domains). A 200-file PR might be split by files, then by security and performance.
A good subtask states its objective, inputs and relevant context, boundaries and exclusions, available tools and permissions, expected output schema, completion test, budget and deadline, and dependencies.
Poor subtask
“Research storage systems”
Better subtask
“Identify storage-related papers published in 2026; return title, date, system contribution, evaluation scale and primary-source URL; exclude database query optimization”
Granularity is a trade-off: tasks too large leave little parallelism, tasks too small let scheduling and communication dominate. T(multi) ≈ T(planning) + T(critical path) + T(communication) + T(merge). Parallelism helps only when the sequential work saved exceeds planning, communication and integration costs; for example, 20 one-line lookups batched into 4 subtasks.
Decomposition can be static (known before execution, such as summarizing five PDFs) or dynamic (a dead link spawns a lookup task), recursive (a review agent spins up one subagent per subfield) or opportunistic (an idle agent picks up the next unlabeled document). Dynamic decomposition needs spawning limits, budgets, depth limits and termination conditions, or delegation explodes recursively.
Common failure
Example
Overlap
Two agents summarize the same paper
Gap
“2020–2023” plus “2025–2026” skips 2024
Hidden dependency
Pricing needs sizing, found only at run time
Unmergeable outputs
One agent returns free text, another JSON
Lost global context
Every section is fine; the report contradicts itself
Straggler
Six agents finish in 20 s, one takes 4 minutes
Communication
Dimension
Options
Rule of thumb
Topology (who talks to whom)
Supervisor–worker, peer-to-peer, mesh
Mirror the task’s dependencies
Medium (how information is exchanged)
Message passing (queue or bus), shared memory (such as a task list), files and artifacts (including databases)
Natural language is flexible but ambiguous; structured messages are easy to validate but less expressive. A practical design uses both, with fields such as task_id, type, status, claim, confidence, evidence and artifact. Separate control from content: operations the runtime knows are DELEGATE, QUERY, REPORT, HANDOFF, UPDATE, CANCEL, ACK, ERROR and COMPLETE, so the runtime can route, retry and stop without asking an LLM what a message meant.
Communication is a hard distributed systems problem. Agent A tells B to “purchase 8 GPU instances” and the network times out with no reply. Did B execute? Retry or give up? At-most-once or at-least-once? Is exactly-once even available? Use an idempotency key on the task_id, plan a compensation or rollback path, and decide who pays for a double purchase.
Multi-agent systems inherit the problems of distributed systems: concurrency and non-determinism, unreliable networks (delay, drop, reordering, duplication), no global clock, partial failure (a slow agent looks like a dead one), consistency among replicated state, and debugging across logs with clock drift. They add scheduling (who runs next, and for how long), resource allocation (models, CPU) and state management (checkpoints, memory, provenance, versioning), with no predetermined request flow. Black-box LLMs make it harder still:
Problem
What happens
Information loss
Summaries drop what the worker knew
Context pollution
What matters gets buried
Stale information
B still acts on X=3 after X=5
Conflicting beliefs
A says safe, B says vulnerable
Lost ownership
Each agent thinks another owns it
Communication loops
A asks B, B asks C, C asks A
Opaque failure
Why did it fail?
A2A and MCP
A2A is the agent-to-agent protocol. Each A2A server publishes an Agent Card at /.well-known/agent-card.json with its identity, skills, endpoints, capabilities (such as streaming) and auth scheme. Three abstractions: a message (one turn between client and agent), a task (a stateful unit of work with an ID and a lifecycle) and an artifact (the task’s output). Follow-ups join the same task. A task moves from submitted to working, can pause for input or auth and resume, and ends completed, failed, rejected or canceled. Callers track long tasks by polling (short jobs), streaming (interactive output) or push notification (hours-long jobs). A2A standardizes communication, not orchestration: delegation, splitting and stopping stay with the harness.
01Submitted
→
02Working
→
03Pauses for input or auth, then resumes
→
04Completed · failed · rejected · canceled
MCP
A2A
Primary abstraction
Capability / tool
Agent / task
Relationship
Agent to service
Agent to agent
Discovery
Tools, resources, prompts
Agent capabilities
Unit of work
One tool invocation
Long-running task
Internal state
Capability interface
Ideally opaque
Analogy
Device or API interface
Service-to-service RPC
One A2A remote agent may itself call twenty MCP servers. In MCP the host, not the model, decides what is exposed.
Design principles
01Ask whether multiple agents are necessary: when one agent is enough, more usually add only overhead
02Decide the topology from task dependencies and minimize communication
03Identify information-loss points: aggregation nodes are where detail drops
04Identify failure-containment boundaries so the system does not stall
A dense mesh is rarely the best default. More communication can raise context pressure without adding useful information.
Self-check
Which architecture suits each kind of task?
Separable subtasks that can run in parallel
Clear stages in a fixed order
Judgment calls
Dynamic allocation to the cheapest capable agent
Self-check
One agent covers 2020–2023, another 2025–2026, and 2024 is missed. Which decomposition failure is this?
06LECTURE 6
Agent design III: memory and reliability智能体设计 III：记忆与可靠性
Key points
Every stage of memory is a policy decision: whether to write, what form, where to store, when to retrieve, how to generalize, when to forget.
Memory needs valid time and provenance, and forgetting is a first-class operation.
At 95% success per step, only 8% of 50-step runs are error-free; errors compound and propagate.
Faults originate in the task specification, harness, model, tools and environment, and each has its own fix.
Six questions for agent memory
01Write or skip: is it worth keeping?
→
02Encode and organize: what should it become?
→
03Store: where should it live?
→
04Retrieve: does this task need it?
→
05Reflect: what broader lesson follows?
→
06Keep, update or delete: is it still valid?
Writing. Writing every interaction raises recall and noise; selective writing raises precision. Triggers can be an explicit user request, deterministic rules, a small classifier or a periodic LLM review.
Organization
Example
Trait
Raw events
“We moved serving from vLLM to SGLang.”
High fidelity; large and redundant
Unstructured summary
“Uses SGLang for model serving.”
Compact, may lose context
Structured state
serving_engine = SGLang
Queryable; the schema constrains
Knowledge graph
Project → USES → SGLang
Linked entities and change over time
Backend
Examples
Good at
Plain files
CLAUDE.md, scratchpad
Easy to inspect, diff and search
Key-value or relational database
Redis, SQLite, Postgres JSONB
Exact lookup, filtering, transactions
Vector store
pgvector, Chroma, Pinecone
Exact or approximate similarity search
Graph database
Neo4j, FalkorDB (Graphiti)
Relationship queries and multi-hop traversal
When memory is used
How
Strength
Weakness
Automatic
Search, LLM, add on every turn
Predictable and simple
Adds latency and irrelevant context
Runtime-controlled
Search or add only when a condition holds
Selective and efficient
Needs rules or a classifier
Model-controlled
Search, add, update and forget as tools for the LLM
Demand-driven
May miss useful reads or writes
Many systems mix them, and read and write policies may differ.
Retrieval. The basic pipeline is query, vector search, top-k, prompt. Semantic match is one signal; the others are scope (same user, project, entity or task), importance (the cost of missing or including it), validity (true for this task at this time) and provenance (source, confidence, evidence).
Reflection. Questions about vLLM, SGLang, TRT-LLM and LLM scheduling suggest the hypothesis “frequent interest in LLM inference systems, medium confidence”. Consolidate after enough new evidence (every N events or at context fill), when facts conflict or repeat, and in a background job when possible.
Forgetting is a first-class operation. Reasons: stale, incorrect or conflicting information, privacy and security, storage and retrieval cost. Controls: mark invalid, erase permanently, expire after a TTL, create tiers and evict from active memory. Memory needs valid time and provenance: “serves models with vLLM” held in April 2026 and was superseded in June by “serves with SGLang”, sourced to episode 28.
Ownership. Agent-owned memory is files in the harness (CLAUDE.md, a scratchpad, task notes) that the harness persists to disk. A dedicated memory system is a service the agent calls, which makes memory an API shared across agents and sessions.
Memory system
Kind
Input and retrieval
Mem0
Fact-centric
Scoped by user, agent or run; filtered vector search, optional graph memory
Graphiti
Temporal knowledge graph
Entities and time-aware facts from episodes; hybrid semantic, keyword and graph search
supermemory
Profile, memory and documents
A user profile kept alongside memories and documents, linked to source and time
Failure mode
What happens
Retrieval miss
Relevant memory exists but isn’t found
Context pollution
Too many irrelevant memories enter context
Memory hallucination
An inference stored as fact: “studying at Harvard” becomes “visiting student at Harvard”
Staleness
An old fact overrides newer information
Isolation failure
Memory crosses a user boundary
Overgeneralization
One event becomes a durable preference
Persistent prompt injection
Untrusted content returns later as instructions
Unbounded growth
Size grows with lifetime, so retrieval cost and quality degrade
Memory is evaluated on retrieval (precision, recall), writes and updates (precision, recall, stale-memory rate), governance (deletion compliance, scope isolation) and system cost (latency, token overhead, store growth). End-to-end benchmarks mainly test recall and reasoning over user–assistant histories, covering recall, temporal reasoning, updates, preferences and abstention: LoCoMo (ACL 2024, 10 long dialogues, 1,540 questions), LongMemEval (ICLR 2025, 5 memory abilities, 500 questions) and BEAM (ICLR 2026, up to 10M tokens, 2,000 questions). The memory stack takes two forms: a separate memory layer (an agent runtime, memory management such as Mem0 or Letta, and vector, SQL, graph or object storage), or memory inside the harness (managed by the runtime, stored as Markdown files).
Reliability: errors compound
At 95% per step, only 8% of 50-step runs are error-free
Chance a run is error-free, %, by number of steps, assuming independent steps
95%1 step
77%5
60%10
36%20
8%50
Lecture 6 slides (0.95 to the power n)
A run succeeds only if every step does, and error propagation makes real runs worse than this. Faults originate in five places: the task specification, the harness, the model, the tool layer and the environment.
Failure
Example
Fix
Ambiguous goal
“Fix the login bug”; fixed the wrong one of three
Name the file, symptom and repro
Proxy objective
“Make the tests pass”; deleted the failing test
Forbid test edits; review the diff
Unstated constraint
“Speed up search”; broke the public API
State what must not change
No definition of done
“Clean up the parser”; never stopped refactoring
Give an exit test and a stop rule
Conflicting goals
“Minimal diff, every edge case”; silently dropped edge cases
Rank the goals explicitly
Failure
Example
Fix
Context truncation
Trimmed the oldest turns and forgot “keep the API”
Pin key rules outside the window
Memory error
Recalled a stale note as fact; edited a moved file
Date memories; re-check before use
State corruption
Kept a half-applied edit; later steps built on broken files
Checkpoint; roll back failed steps
Control loop
No step or cost budget; 200 steps, no progress
Cap steps, cost and time
Blocking fan-out
Waited on every subagent; one slow child stalled the run
Set timeouts; use partial results
Failure
Example
Fix
Hallucination
Called a nonexistent requests.fetch()
Check calls against real docs
Instruction violation
Changed a signature it was told to keep
Restate constraints; check the diff
Faulty reasoning
“A timeout means the network is down”; added retries to a deadlock
Test the hypothesis before fixing
Reasoning–action mismatch
“I’ll only touch parser.py”; rewrote three other files
Compare each action to the plan
Unverified success
“Tests probably pass now”; ran nothing
Require test output as proof
Long-horizon degradation
Undid its own fix from step 12
Re-read state; split into short tasks
Failure
Example
Fix
Truncated output
First 100 lines of a log; missed the error on line 4,000
Return errors first; allow paging
Silent failure
Exit 0, empty result; assumed no matches
Separate “empty” from “failed”
Ambiguous schema
A date with no time zone; job scheduled 8 hours off
Typed, explicit schemas
Flaky call
A timeout on the first try; declared the service down
Retry with backoff before judging
Misleading error
“Permission denied” on a typo; retried with sudo
Precise, actionable errors
Failure
Example
Fix
Stale state
Yesterday’s cached build; tests passed on old code
Build clean from a fresh cache
Version mismatch
Python 3.8, not 3.12; shipped code that crashed
Pin versions in a lockfile
Random failure
Failed because of a leap year
Rerun failures; quarantine flakes
Changed page layout
A redesigned checkout page; clicked the wrong button
Re-read the page before acting
Rate limit
HTTP 429 after 50 calls; stopped mid-migration
Throttle and checkpoint long jobs
Specification: write “done” as a testAcceptance tests up front; list what must not change
Harness: bound every runStep, cost and time caps; checkpoint and roll back
Model: demand evidenceProof for every claim; verify; use stronger models
Tool: fail loudlyTyped schemas; errors distinct from empty results
Environment: pin and isolateFresh, pinned sandboxes; retry flakes before judging
We do not yet understand agents at scale, and there is little formal best practice.
Self-check
Where did each fault originate?
Told to “make the tests pass”, it deleted the failing test
No step or cost budget: 200 steps, no progress
“Tests probably pass now”, with nothing run
A date with no time zone: a job 8 hours off
Failed because of a leap year
Self-check
At 95% success per step, independent steps, how likely is a 20-step run to be error-free?
0.95 to the 20th power is about 36%; 5 steps about 77%, 10 about 60%, 50 about 8%.
07LECTURE 7
Agent design IV: evaluation智能体设计 IV：评估
Key points
The object of evaluation moved from the base model to model, prompt, tools and harness together, graded by tests and environment state.
Eight representative benchmarks each test one environment: Q&A, function calls, the web, the desktop, a repository, customer service, a terminal and a simulated company.
It is hard because leakage has four layers, single-turn tasks reward guessing, and success rate alone ignores cost and safety.
Benchmarks saturate ever faster; a real agent is judged on capability, cost and speed together.
History and objectives
An agent can be evaluated by component (planning and decomposition; memory extraction, consolidation and retrieval; tool discovery, selection and parameterization; error recovery) or end to end (success rate, cost per task, time to finish). Benchmarks are moving toward larger scope, less specific instructions, more realistic use and newer problems.
MMLU-Pro, LiveBench, LiveCodeBench, FrontierMath, Humanity’s Last Exam
Difficulty, robustness, freshness
2025–2026
GDPval, BrowseComp, Terminal-Bench, ARC-AGI-3
Larger scope, more realistic use
Objective
Representative benchmarks
What success means
Predict text
PTB, WikiText
High probability on observed text
Produce a correct answer
MMLU, GSM8K, GPQA, HLE
Match a known answer
Produce a preferred response
Arena, AlpacaEval, MT-Bench
Favorable human or model judgment
Produce functioning code
HumanEval, LiveCodeBench, SWE-bench
Pass executable tests
Complete an interactive task
WebArena, OSWorld, τ-bench, Terminal-Bench
Reach the required environment state
Metrics: earlier benchmarks used perplexity (exponentiated average negative log-likelihood on held-out text, lower is better), accuracy or F1, and preference win rate. Capability uses pass@k (at least one of k attempts passes) and pass^k (all k pass, introduced by τ-bench). Cost is mostly LLM cost; tool calls and cache misses are often overlooked. Latency is not only the LLM’s.
The benchmark loop: for each task, reset the environment to the task’s initial state; while not terminated and within budget, observe, let the agent pick an action from the instruction, observation, history and tools, and execute it; finally an evaluator scores the final answer, the final environment state and the trajectory.
Eight representative benchmarks
Benchmark
Agent operates in
Typical actions
Graded on
Notes
GAIA
Web, file and code tools
Search, inspect files, calculate
A short final answer
466 human-written questions in 3 levels (≤5 steps, 5–10 steps, arbitrarily long); tests the whole agent system; does not test actions such as sending email or booking
BFCL
Function and API interface
Structured function calls
Calls, arguments and resulting state
AST or execution checks; typical errors are invented IDs (should ask the user) and irrelevant calls; read category scores, not the composite
WebArena
Self-hosted websites
Click, type, move between pages
Website or backend state
812 tasks across shopping, admin, Reddit, GitLab, maps, Wikipedia; accessibility-tree observations; WebArena-Verified drops the LLM judge for deterministic checks on HTTP traces
OSWorld
A full desktop VM
Mouse, keyboard, apps
Files, app and system state
369 tasks with screenshot observations; OSWorld 2.0 (2026) has 108 long workflows, often over an hour, with about 27 checkpoints for partial credit
SWE-bench
A software repository
Inspect, edit, run tests
Whether the patch passes tests
Variants: Verified (human-checked), Lite (300 tasks), Multilingual (9 languages), Multimodal, Pro (harder, multi-file)
τ-bench
Customer-service APIs and dialogue
Talk to a user, call APIs
Database state and the reply
An LLM simulates the user; the agent is bound by airline policy; reward is action × output; read-only calls may vary but unintended writes fail
Terminal-Bench
An isolated terminal container
Arbitrary shell commands
Task-specific verifier tests
Programming, sysadmin, data processing, model training, security, science; 89 tasks in TB 2.0, 66 in TB 4.0 after pruning saturated ones; 8-hour timeout
TheAgentCompany
A simulated software company
Web apps, code, files, talking to coworkers
Workflow checkpoints and final state
175 tasks across software, PM, data science, HR, finance; partial score = 0.5 × earned/total + 0.5 × full completion
Terminal-Bench sits between SWE-bench and OSWorld: the agent acts through shell commands in a container and sees terminal output, on tasks that configure, debug, repair and build, with no mouse-coordinate grounding, so it tests long-horizon reasoning and system interaction. pass^k is much stricter than pass@k: all k independent runs must succeed, not one.
Why evaluation is hard
Training leakageThe task or its human fix was in the training data
Eval-time leakageThe agent searches the web and copies the future fix
Environment leakageThe sandbox exposes future commits or hidden tests
Benchmark overfittingPrompts and scaffold tuned on the public test set
Agent performance is roughly a function of the LLM, prompt, scaffold, tools, retriever, memory and environment, and any of them can carry benchmark information: solutions used as fine-tuning trajectories, answers indexed in a RAG corpus, earlier attempts kept in memory, researchers tuning on test failures. Saying “the model was not trained on the test set” is nowhere near sufficient. Realistic tools are required for agent evaluation, and they widen the leakage surface.
A single user turn. Most benchmarks give one fully specified instruction: prompt in, final state out, no user in the loop. Real work is ambiguous, revised and interrupted, and single-turn scoring rewards confident guessing because a clarifying question costs steps and earns nothing. Worth testing: does the agent ask when the goal is underspecified, re-plan after a mid-task correction, and carry constraints across turns?
Other challenges: success rate alone cannot tell a 200-step, $40 run from a 12-step, $0.30 run; non-determinism makes single-run scores noise; nothing penalizes deleted files, sent emails or money spent on the way to a pass; outcome-only scoring passes lucky or unsafe runs, while trajectory matching fails valid ones.
The frontier reaches each baseline faster
Months from a benchmark’s release to the frontier reaching the stated baseline
GLUE (2018) · human level 87.114 mo
MMLU (2020) · expert level 89.839 mo
GPQA Diamond (2023) · PhD experts 69.710 mo
SWE-bench Verified (2024) · 33% → 95%22 mo
ARC-AGI-2 (2025) · average human 6011 mo
ARC-AGI-3 (2026) · 0.5% → 62.7%6 mo
Lecture 7 slides
Benchmarks saturate quickly. As of September 2026, SQuAD, GLUE, SuperGLUE, MMLU, MATH, HumanEval, Chatbot Arena, MT-Bench, WebArena, SWE-bench Verified, GPQA Diamond, MMMU and OSWorld are saturated; FrontierMath, Humanity’s Last Exam, ARC-AGI-2 and 3, Terminal-Bench, GDPval, SWE-bench Pro, Remote Labor Index and OSWorld 2.0 remain open. Saturated means the frontier reached the human baseline or top models stopped separating.
Dimension
Earlier evaluation
Current frontier
Object evaluated
The base model
Model plus prompt, tools and harness
Interaction
Static, single-turn
Stateful, long-horizon
Grader
Exact match or likelihood
Tests, environment state, humans or judge models
Dataset
Fixed and public
Fresh, hidden, dynamic or contamination-aware
Main metric
Average accuracy
Reliability, quality, cost and latency beside success
Primary concern
Generalization across examples
Robust completion under realistic constraints and safety requirements
A real-world agent is judged on capability, usability, cost and speed together. The slides rank by Pareto dominance: A dominates B if A is no worse on any metric and better on at least one, and rank 1 is the Pareto frontier.
Self-check
A task runs 4 times independently and succeeds once. What are pass@4 and pass^4?
pass@k needs at least one of k runs to pass; pass^k, introduced by τ-bench, needs all k to pass and is far stricter.
Cost is users × sessions × turns × steps × tokens × price; the middle three shrink with shorter trajectories, the price with caching, batching and routing.
Cache reads are most of the tokens, and every token left in context is paid for again on every later step.
Every saving has a counter-cost: measure the net effect on the same tasks and cap each task’s budget.
Completion time is the critical path; harness architectures run from monolithic to modular to microkernel as responsibility moves from code to services.
01Turns per task
→
02Steps per turn
→
03Tokens per step
→
04Price per token
Fewer turns and steps
This lecture covers LLM cost only, not other compute costs.
Turn k re-reads turns 1 to k−1, so total cost grows with the square of the turn count. In the course’s own Claude Code logs, single-prompt sessions were 68% (assignment 1) and 75% (assignment 2), and a few marathon sessions ran past 100 turns. Users come back because there was no plan, the agent stopped early or it never checked. The remedies are confirming goal and plan once before acting, splitting into subtasks that each carry a done check, and self-verifying before handoff, so the task finishes with no user check-ins.
Most turns take few steps (largely because effort was set to medium or high), but some ran almost 1,000. Every model call re-reads the full context, so fewer calls cost less: batch independent calls into one step (read five files in one step, not five), reuse plans and skills that worked, and load the files you need at the start. Related work covers skill libraries (Voyager, SkillRL), insights from experience (ExpeL, ReasoningBank), plan-template caching and durable conversational memory (Mem0, LightMem, SimpleMem). Reuse pays when tasks recur; memory upkeep and stale entries can erase the savings.
Fewer tokens per step
97.6%Share of tokens that were cache reads, assignment 1
208K / 301KMain-agent median context per step (assignments 1 / 2)
Output was about five times smaller than fresh prefill (cache writes), and fresh prefill about a hundred times smaller than cache reads. Context grows fast, reaching 650K to 670K tokens around step 500. Tokens that stay in context are paid for again on every later step.
Reduce input
How
Drop what is stale
Truncate stale results; compact the history; mask tool outputs already used; summarize old turns into state
Shrink what enters
Compress observations, keeping errors rather than the full log; save big outputs to files and pass a path
Load only what is needed
Retrieve selectively, with only relevant tools and memory; isolate exploration in a subagent that returns a summary
Related results: ACON cut peak context by 26% to 54%; LLMLingua reached up to 20× fewer tokens, though prompt compression pays off only on long prompts; MemGPT treats context as fast memory; filtering MCP results in code took 150K tokens to 2K; tool search cut tool tokens by 85% and programmatic tool calling by 37%. TRACE found that summaries cause rework.
For output: state length, format and what to skip in the prompt (“answer in 100 words or fewer”, “return only JSON”, “don’t restate the problem”); set max_tokens in the API (it can cut an answer mid-sentence, and reasoning can use up the cap); require a schema with only the needed fields (restrict enums and optional fields; prose around a tool call is pure cost); and match reasoning effort to the step (low for lookups and formatting, high for planning and debugging). Strict formats can hurt reasoning.
A lower price per token
Batching plus caching brings input to 5% of the base price
Price per token relative to base input, Claude API
Output token5×
Base input token1×
Batch input0.5×
Cached input0.1×
Batch plus cached input0.05×
Lecture 8 slides
Keep prefixes stable (system prompt, tools, early history) and batch what can wait (evals, backfills, bulk jobs). Model routing comes in three forms: choose up front (a router trained on preference data, which can also pick agent roles, as in RouteLLM and MasRouter), escalate on failure (a cheap model first, a larger one when a check fails; checks can be a learned scorer, self-verification or execution feedback, as in FrugalGPT, AutoMix and EcoAssistant) and re-route each step (on history and budget, with an orchestrator calling models as tools, as in ToolOrchestra). The hard part: one cheap planning mistake can waste many later actions. System techniques for lowering price come in later lectures: MoE, quantization, pruning, distillation and workload-targeted fine-tuning; speculative decoding, KV-cache optimization and better scheduling; faster kernels and specialized accelerators.
Every saving has a counter-cost
Technique
Counter-cost
Compaction
Dropped files get re-read, and the summary rewrites the cache
Routing
A bad plan from a cheap model adds steps or a failed attempt
Subagents
Each subagent re-reads the shared files and instructions
Speculation
Wrong guesses burn tokens and contend for the same tools
Net saving = gross saving − counter-cost, measured on the same tasks. Cap each task’s budget (illustrative limits: $10, 80 model calls, 15 minutes of tool time, 2 retries per step) and stop when the verifier passes, no new test passes in N steps, the same error repeats, or a limit hits, handing off with a summary. Limits can be set from traced runs, for example at twice the p95 of successful tasks. A stopped task with a summary costs less than a runaway one.
01Same steps every time: have the LLM write the script once, then just run it
02Fixed input format: parse with regex, parsers and templates at no cost per call
03Narrow label or score: train a small classifier
04Fixed workflow with little reasoning: script the workflow and call the LLM only at the steps that need it
Making the agent faster
Completion time is the critical path, not the sum of every step. Faster is usually cheaper, but not always: small models may take more steps, and parallel work spends more tokens.
Speed up LLM servingEvery cost technique above also cuts latency: fewer turns, steps and tokens, cheaper serving
Improve orchestrationAsynchronous execution (start the next step while a tool runs), persistent sessions (one shell or browser across steps), streaming and pipelining (start the tool call before the response finishes), colocated services
Speed up tool callsBuilds, tests and page loads can outlast the model call: cache tool results, run heavy compute on a server, use faster tools, time out hung calls; measure tool time per step before tuning the model
ParallelizeIssue independent tool calls in one step; split independent subtasks across parallel subagents and merge their summaries; shared state needs a merge step
Speculative actionsA cheap model predicts the next action, which runs while the main model thinks; a match skips the wait, a miss costs one discarded call; it pays when guesses are often right and actions can be undone
What is speculated ranges from the next tool (SPORK) and the tool with arguments (PASTE) to any action, several branches (B-PASTE) and a whole executed chain. Predictions come from a draft model, from an early fork of the same model, or from history (PASTE sees that a search is usually followed by a visit and fills the URL from the result to run it early). The open question: how to schedule speculative calls under limited GPU, CPU and API budgets so that the critical-path time saved exceeds the work wasted.
Optimization is a loop: run and log every call, token, tool and wait; measure where cost and latency come from; change one thing (compaction, routing, caching or parallelism); re-evaluate success, spend and p95 latency on the same tasks; repeat.
Three harness architectures
Dimension
Monolithic
Modular
Microkernel
Design principle
Everything in one runtime
Interfaces, one controller
Minimal core plus services
Replacing a component
Difficult
Easy (Docker to Firecracker is one provider swap)
Easy, if protocols match
Execution isolation
Optional
Optional
An architectural emphasis
Deployment
Usually together
Together or separate
Often independent
Independent scaling
Limited
Possible
Strong
Communication cost
Low
Low to medium
Potentially higher
Typical use
Standalone agents
Agent frameworks
Multi-agent infrastructure
Strengths
Simple to build and debug; easy shared state
Clear responsibilities; testable; third parties plug in
The controller still owns the lifecycle; coordination grows complex
More failure modes, harder end-to-end debugging, needs standard protocols
Running one Python snippet: the monolith executes it locally, with the fewest hops but weak isolation; the modular harness calls SandboxProvider.run(code), so you swap the provider, not the agent; the microkernel sends a request to an execution service that owns allocation, isolation and sandbox lifecycle. Responsibility moves from code to interfaces to services.
What is unique in eight harnesses
Harness
Profile
Distinctive design
Claude Code
Anthropic, TypeScript, closed
A policy layer and typed hooks around Claude
Codex CLI
OpenAI, Rust
Sandbox before approval: commands run in a sandbox with no network, and the agent asks only to leave it; a Lark grammar sets the patch envelope, and hunks are matched in four passes
DeepSeek Harness
TypeScript
Even the loop is a plugin, replaceable through YAML layers; each request is a frozen view of the session log, to which the harness only appends; compare-and-swap edits fail if the file changed since the model last read it
OpenCode
TypeScript, Bun
The harness is a server: the TUI, SDKs and web app share one OpenAPI HTTP server; nine replacers go from exact to anchor match and reject fuzzy matches that are too large; a shadow git repository snapshots each step so /undo restores files
Pi
TypeScript
A minimal core of four tools, with no MCP, subagents or approvals; cache warming only when the expected saving is $0.05 or more; pi-ai supports 42 providers on 10 wire APIs
oh-my-pi
TypeScript plus Rust
Hashline edits: a 4-hex file tag plus line numbers, so the model never retypes old text; one shell stays open, with native search, AST, LSP and a debugger; snapcompact turns old history into images
OpenClaw
TypeScript
A gateway for 27 chat channels with one queue lane per session; before compaction a silent turn writes dated notes, and recall combines full-text and vector search; fences isolate untrusted text, a network read taints the turn, and secret sentinels are monitored
Hermes Agent
Python
A background fork examines nudged turns and edits memory and skills; compaction starts at 50% of the window, the earliest of the eight; tool-call rows are written to the session database before any side effect
Self-check
Which saving carries each counter-cost?
Dropped files get re-read; the summary rewrites the cache
A cheap model’s bad plan adds steps or a failed attempt
Each one re-reads the shared files and instructions
Wrong guesses burn tokens and contend for tools
Net saving is gross saving minus counter-cost, measured on the same tasks.
Self-check
On the Claude API, batched and cached input costs about what share of the base input price?
Batch input is 0.5×, cached input 0.1×, and together 0.05×; output tokens are 5×.
09LECTURE 9
LLM basics: the life of a request and the transformerLLM 基础：一次请求的生命周期与 Transformer
Key points
A request passes through six stages: tokenize, schedule, prefill, decode, detokenize, stream.
Prefill is compute-bound and sets time to first token; decode is bandwidth-bound and sets tokens per second.
KV-cache size per token caps concurrency; GQA brings Llama 3 70B to 320 KB per token.
In MoE, memory scales with total parameters and compute with active ones.
01Tokenize
→
02Schedule
→
03Prefill: time to first token
→
04Decode and sample: tokens per second
→
05Detokenize
→
06Stream
Six stages
Tokenize. Text is split into tokens, not words, at about 3 to 4 characters per token: “Serving is hard” is four tokens. Token count drives cost and memory, and tokenizers differ across models. Byte-level BPE starts from the 256 byte values and repeatedly merges the most frequent adjacent pair, so there are no unknown tokens. A bigger vocabulary means fewer tokens but a larger embedding table.
Schedule and batch. Requests arrive independently and GPUs want batches. Continuous batching lets requests join or leave at any step, with no wait for a full batch; admission is capped by KV-cache memory.
Prefill. The whole prompt runs in one pass with all prompt tokens attending in parallel. It is compute-bound, its cost grows with length, and it builds the KV cache and emits the first token.
Decode and sample. One forward pass per new token, and the cache grows by one token per step. It is bandwidth-bound, so batch it; when the cache is full, preempt or queue. Sampling picks the next token from the logits, and its settings change the answer and how long the request runs (table below).
Detokenize and stream. Token IDs become text as they are produced, with partial characters buffered. A stop token, length limit or disconnect ends the stream, and stopping early frees cache memory.
Sampling setting
Example
Effect
temperature
0
0 always takes the top token; above 1 widens the choice
top-k
50
Samples from the k most likely tokens; k is fixed
top-p
0.9
The fewest tokens summing to p; adapts to confidence
The transformer
01Token IDs
→
02Embedding: each ID becomes an 8,192-dim vector
→
03Transformer block × 80: self-attention plus FFN
→
04LM head: one score per vocabulary token
→
05Softmax: next-token probabilities
A model is a stack of identical transformer blocks. Integer token IDs enter an embedding that turns each into a vector (8,192 dimensions in Llama 3 70B); they pass through L blocks (80 in Llama 3 70B), each with self-attention (mixes information across tokens) and a feed-forward network (transforms each token on its own); an LM head scores every vocabulary token, and softmax turns scores into probabilities.
Each block is pre-norm with a residual stream: RMSNorm, then attention, added back to the stream; RMSNorm, then the FFN, added back again. RMSNorm rescales by the root mean square with one learned gain per dimension; pre-norm keeps deep stacks stable in training; the residual stream flows through all 80 blocks, and each sublayer only adds an update.
Self-attention lets each token read every earlier token. Each token emits a query Q, key K and value V (X times learned weight matrices). Q·K divided by √dₖ decides how much to read from each token; the scaling stops softmax from saturating, because the dot product grows with dₖ. A causal mask puts −∞ above the diagonal to hide future tokens, softmax makes each row sum to 1, and the output is a weighted mix of the rows of V. n tokens give n² scores, about 4n²d FLOPs per layer. Storing the score matrix S for 8,192 tokens, 64 heads and 80 layers would take about 690 GB; FlashAttention computes softmax tile by tile and never stores S.
Step
Formula
Shape
Input
X
n × d
Queries
Q = X W_Q
n × dₖ
Scores
S = Q Kᵀ ÷ √dₖ
n × n
Weights
A = softmax(S + M)
n × n
Output
O = A V
n × dᵥ
Multi-head attention splits Q, K and V into 64 heads of 128 dimensions, runs masked attention in each, and mixes them with Wₒ. Since 64 × 128 = 8,192, the FLOPs equal one wide head: splitting changes what is learned (some heads attend to the previous token, the first token, nearby tokens, the whole context), not the cost.
The KV cache and shared KV heads
KV cache per token = 2 × layers × KV heads × head dimension × bytes per value. In MHA each query head has its own KV head; GQA lets groups of query heads share KV heads; MQA shares one KV head across all. With 8 query heads, their KV caches are 1×, ¼× and ⅛×. Llama 3 70B’s 64 query heads share 8 KV heads: 2 × 80 × 8 × 128 × 2 B = 320 KB per token, against 2.5 MB with MHA.
Llama 3 70B
Value
Model width d
8,192
Layers L
80
Query heads H and head dimension dₖ
64 and 128
KV heads
8
FFN hidden width
28,672 (3.5 × d)
Vocabulary V
128,256
Most of Llama 3 70B’s parameters sit in the FFN
Parameters, billions, by component
Attention (80 layers)12.1B
FFN (80 layers)56.4B
Embedding and LM head2.1B
Meta, The Llama 3 Herd of Models (2024); derived in lecture 9
Attention takes about 151M parameters per layer and the FFN about 705M; 80 layers make 68.5B, and with the embedding and LM head the total is 70.6B. The attention and FFN totals in the chart are the per-layer figures times 80.
Systems intuition
141 GBWeights (BF16, 2 bytes per parameter)
141 GFLOPsCompute per token (about 2 × parameters)
320 KBKV cache per token
On two H200s (1,978 TFLOP/s, 9.6 TB/s and 282 GB in total), an 80,000-token prompt takes about 9.95 s to prefill and 167 GB of memory for one request, and 637 s and 1,779 GB for 64 requests. Compute grows with n² and the KV cache linearly with n, which is what makes long sequences hard.
MoE: memory scales with total parameters, compute with active ones
MoE replaces each block’s FFN with a router and a set of smaller expert FFNs; only a few run per token (for example 2 of 8), and their outputs are summed with weights.
Model
Experts
Total
Active per token
Weights (BF16)
Llama 2 70B
All (dense)
69B
69B
138 GB
Mixtral 8x7B
2 of 8
46.7B
12.9B
93 GB
Llama 3.1 405B
All (dense)
405B
405B
810 GB
DeepSeek-V3
9 of 257
671B
37B
1,342 GB
DeepSeek-V3 activates only 37B parameters per token
Active parameters per token, billions; GFLOPs per token are about twice this
Llama 2 70B69B
Mixtral 8x7B12.9B
Llama 3.1 405B405B
DeepSeek-V337B
Jiang et al. 2024; DeepSeek-AI 2024; Touvron et al. 2023; Llama Team 2024; cited in lecture 9
Same quality, less compute: Mixtral matches Llama 2 70B at about a fifth of the FLOPs, and DeepSeek-V3 rivals Llama 3.1 405B at about an eleventh, but needs 1.7 times the memory.
Further: eight advanced variants
Variant
How
Serving effect
Sliding window
Local layers read only the last w tokens; a few global layers keep long-range recall (Gemma 3: 5 local to 1 global, w = 1,024; Mistral 7B: all local, w = 4,096)
At 128K context, Gemma 3 stores about 6× less KV than all-global layers
Multi-latent attention (MLA)
Cache one compressed latent per token and rebuild K and V when needed
DeepSeek-V2 cuts KV cache 93% against DeepSeek 67B and lifts generation throughput 5.8×
Linear attention
A fixed-size state updated in place; Gated DeltaNet fades the old state, then overwrites (Qwen3-Next: 36 GDN plus 12 attention layers; Kimi Linear and Qwen3.5 use the same 3:1 recipe)
Three of four layers hold a constant-size state per request at any length
State-space models
A small state per channel whose decay each token sets (Mamba’s selectivity; Mamba-2 turns the scan into matmuls)
No growing KV cache; Mamba reports up to 5× a transformer’s generation throughput
Sparse attention
A light indexer scores past tokens and each query reads only the top 2,048 (DeepSeek-V3.2)
Work per token is capped, but every token’s KV is still stored
Looped transformers
One block applied K times
Depth costs compute and latency, not weights
Multi-token prediction
Predict several tokens at once
Listed by name only in the slides
Diffusion LMs
Fill masked tokens over a few parallel refinement steps (LLaDA 8B, Dream 7B)
One pass fills many tokens, but fewer steps cost quality
Self-check
Which stage is bandwidth-bound and sets tokens per second?
Prefill runs the whole prompt in one compute-bound pass and sets time to first token; decode runs one pass per token and is bandwidth-bound.
Self-check
Llama 3 70B has 80 layers, 8 KV heads, head dimension 128, in BF16. How large is its KV cache per token?
2 × 80 × 8 × 128 × 2 B = 320 KB; with a KV head per query head (MHA, 64 heads) it would be 2.5 MB.
10LECTURE 10
GPU basics: why decode is bandwidth-boundGPU 基础：为什么 decode 受带宽限制
Key points
A GPU trades a few fast cores for thousands of simple ones and hides memory latency by switching warps.
Every kernel is bounded by peak compute, memory bandwidth and capacity; the roofline’s ridge point separates the two kinds of bottleneck.
Batch-1 decode uses 0.5% of an H200’s peak compute; batching helps, but decode attention stays bandwidth-bound.
Compute grows faster than bandwidth each generation; reaching full bandwidth means keeping more bytes in flight.
GPU and CPU
64-core server CPU
NVIDIA H200 SXM
Parallel units
64 cores
132 SMs, 16,896 FP32 cores
Threads in flight
270,336 (2,048 per SM)
Memory bandwidth
about 0.46 TB/s (DDR5)
4.8 TB/s (HBM3e)
Peak dense math
about 5 TFLOPs FP32
989 TFLOPs BF16
Hides latency by
Caches, out-of-order cores
Switching to a ready warp
A CPU is a sports car that carries two riders fastest; a GPU is a bus fleet, slower per trip but moving thousands. LLMs fit GPUs because every token runs the same matmuls on the same weights.
Warp switching hides memory latency. While one warp waits on memory, another issues, so the math units stay busy. On A100, H200 and B200, each SM has four warp schedulers that manage up to 16 warps each, 64 resident warps per SM.
SIMT. One warp instruction drives 32 threads; fetch, decode and scheduling are paid once, the work of 32 scalar instructions.
Branch divergence. When threads in a warp take different branches, the warp runs both paths in turn: f might use 8 of 32 lanes and g the other 24.
The GPU gives up
In exchange it gets
Single-thread speed (modest clock, no speculation; serial code runs far slower)
Area and power spent on thousands of arithmetic units
Implicit parallelism (about 270K threads to fill 132 SMs; small jobs leave the chip idle)
Latency hidden by switching warps, not by caches and speculation
Independent control flow (threads run in groups of 32; divergence costs throughput)
One fetch and decode drives 32 cores
On-chip memory per thread (about 240 bytes at full occupancy)
Every thread’s state stays resident, so warp switches are free
Three limits and the roofline
Memory bandwidth rises each generation, but more slowly than compute
HBM bandwidth, TB/s; Rubin preliminary
2 TB/sA100
4.8 TB/sH200
8 TB/sB200
22 TB/sRubin
Lecture 10 slides (NVIDIA datasheets)
A100
H200
B200
Rubin
Peak BF16 compute (how fast the math runs)
312 TFLOP/s
989 TFLOP/s
2,250 TFLOP/s
about 4,000 TFLOP/s
Memory bandwidth (how fast weights and KV are read)
2.0 TB/s
4.8 TB/s
8 TB/s
about 22 TB/s
Memory capacity (how many weights and KV entries fit)
80 GB HBM2e
141 GB HBM3e
180 GB HBM3e
288 GB HBM4
Compute is the engine, bandwidth the fuel line, capacity the tank. Kernel time ≈ max(bytes ÷ bandwidth, FLOPs ÷ peak).
Decode, batch 1: compute waits
3.3 ms per token
16 GB of weights read from HBM
Each weight used once: 0.016 ms of math waits on 3.3 ms of reads
Tensor cores idle 99.5% of the time
Prefill, 2,048 tokens: compute is busy
33 ms per prompt
HBM idle once weights are loaded
Each weight feeds 2,048 tokens, so math sets the pace
33 TFLOP, near peak
The roofline uses arithmetic intensity, FLOPs per byte moved to or from HBM, to find the limit. The ridge point is peak FLOP/s ÷ bandwidth, about 206 FLOP/B on H200. Left of the ridge a kernel is memory-bound (speed = bandwidth × intensity); right of it, compute-bound (speed = peak).
Operation
Arithmetic intensity (b bytes per element)
Vector add, length N
1/(3b)
GEMV (decode), N × N, each weight used once
about 2/b
Decode attention, each KV entry used once per head
2/b
Square GEMM, each element reused about N times
2N/(3b)
Batch-1 decode is a GEMV at about 1 FLOP/B and reaches about 5 of the H200’s 989 TFLOP/s, 0.5% of peak. To raise it, increase arithmetic intensity: batch more, quantize, shrink KV traffic. Batching lifts weight GEMVs toward the ridge (256 at batch 256), but decode attention stays at 2/b because each request reads its own KV.
With longer context, most bytes read per decode step come from the KV cache
HBM bytes read per decode step, GB; Llama 3 8B, batch 16, BF16; weights fixed at 16.1 GB
18.2 GB1K
24.7 GB4K
33.3 GB8K
50.5 GB16K
84.8 GB32K
Lecture 10 slides (from the Llama 3 paper and H200 datasheet)
The KV cache is 128 KiB per token and grows with batch × length; at 4.8 TB/s a step takes 3.8 ms at 1K and 17.7 ms at 32K. Lower precision pushes the ridge further right: Rubin’s dense NVFP4 needs about 1,600 FLOP per byte to be compute-bound. From A100 to Rubin, BF16 compute grows about 13× and bandwidth about 9×, so kernels are increasingly bandwidth-bound.
Inside the H200
The H200 has 132 SMs that share a 50 MB L2 cache and 141 GB of HBM3e in six stacks at 4.8 TB/s, with 900 GB/s of NVLink to other GPUs. Each SM has four partitions, each with 64 KB of registers, 32 FP32 cores and a tensor core, issuing one warp instruction per cycle; the SM shares 256 KB of L1 and shared memory and keeps 2,048 threads resident.
Tensor cores perform a small matrix multiply-add D = A × B + C (such as 16×16 by 16×8) per instruction: one mma is 2,048 multiply-adds, 64 times an FMA’s work. Matmuls are over 99% of LLM FLOPs; softmax and norms use the FP32 cores. They are fast for four reasons: fewer instructions per FLOP, dedicated multiply-add datapaths, operand reuse (each value of A feeds several output columns, each value of B several rows), and lower precision (FP16, BF16, FP8) that raises throughput and shrinks operands, usually with higher-precision accumulation.
Memory level (H200)
Capacity
Latency (Hopper)
Bandwidth
Registers
256 KB per SM
In the pipeline
Read every instruction
Shared memory / L1
256 KB per SM
About 30 cycles (40 for an L1 hit)
128 B/clock per SM, about 33 TB/s over 132 SMs
L2 cache
50 MB
About 250 cycles
About 12 TB/s on Hopper
HBM3e
141 GB in 6 stacks
About 500 cycles
4.8 TB/s
Host memory and peer GPUs
Microseconds
NVLink 4 450 GB/s each way; PCIe 5.0 about 64 GB/s
Memory is a 250 ns pipe
Little’s law: bandwidth = bytes in flight ÷ latency. A naive kernel keeps about 4 KB in flight per SM and reaches 2.2 TB/s, 45% of peak; filling 4.8 TB/s needs about 9 KB per SM (4.8 TB/s × 250 ns ÷ 132). The hardware fixes the pipe’s length, so the only way to raise bandwidth is to keep more bytes inside it: 16-byte vector loads, four independent loads per thread, and bulk asynchronous copies (TMA).
H200 (Hopper)
B200 (Blackwell)
Rubin
HBM
141 GB HBM3e
180 GB HBM3e
288 GB HBM4
Bandwidth
4.8 TB/s
8 TB/s
22 TB/s
Bytes in flight, GPU
1.2 MB
2.0 MB
5.5 MB
Bytes in flight, per SM
9 KB
14 KB
25 KB
New data-path feature
TMA, clusters
TMEM, 2-SM MMA
Inline TMA descriptors
Further detail: coalescing merges 32 threads’ consecutive 4-byte reads into one 128 B request; 16-byte vector loads let one warp instruction move 512 B; cp.async (Ampere and later) copies into shared memory without staging in registers; Hopper’s TMA lets one thread start a whole tile copy, putting tens of KB in flight from one instruction, and TMA multicast lands one L2 read in the shared memory of every SM in a cluster of up to 8 thread blocks; Blackwell keeps results in 256 KB of tensor memory, freeing registers for more loads, and pairs two SMs on one matrix multiply, each fetching half the shared tile.
From one GPU to many
HGX H200 (8-GPU node)
GB200 NVL72 (rack)
Layout
8 GPUs over NVSwitch; 8 NICs at 400 Gb/s to the scale-out network
Rows of 4 GPUs plus 2 CPUs, 9 rows joined by NVLink switches
HBM bandwidth
4.8 TB/s
8 TB/s
NVLink
900 GB/s
1.8 TB/s
RDMA
50 GB/s
100 GB/s
From G80 (2006) to B300 (2025), peak math grew about 43,000× and bandwidth about 90×. Two decades moved from graphics to general-purpose compute (CUDA, caches, HBM2, NVLink, tensor cores), then to low-precision AI engines (TF32, BF16, FP8, TMA, FP4, TMEM, NVL72).
Further: other accelerators
Chip
Traits
Programming
AMD MI355X
8 compute dies seen as one GPU; 256 MB Infinity Cache; 64-thread wavefronts; 5.0 PF dense FP8, 288 GB, 8 TB/s
ROCm and HIP; PyTorch, vLLM and SGLang run; Triton or Composable Kernel
Google TPU Ironwood
Weight-stationary systolic arrays in a 3D torus of 9,216 chips; 4.6 PF dense FP8, 192 GB, 7.4 TB/s
JAX or PyTorch/XLA; Pallas for custom kernels
Amazon Trainium3
8 NeuronCores per chip, each with tensor, vector and scalar engines; 32 MB software-managed SRAM per core, no hardware cache; 2.5 PF MXFP8, 144 GB, 4.9 TB/s
Neuron SDK; NKI kernels in Python move tiles explicitly; no CUDA
Cerebras WSE-3
A whole wafer as one chip, 44 GB on-chip SRAM, no off-chip memory
Larger models stream weights in
Groq LPU
230 MB SRAM, no off-chip memory
A 70B model at 8 bits needs about 300 chips
Microsoft Maia 200
272 MB SRAM plus 216 GB HBM3e
Available only inside Azure
Self-check
The in-class question: during batch-1 decode, what share of an H200’s peak compute is used?
A batch-1 GEMV does about 1 FLOP per byte, so at 4.8 TB/s it reaches about 5 of 989 TFLOP/s, 0.5%. Raise it by batching more, quantizing and shrinking KV traffic.
Self-check
What limits each operation on an H200?
Square GEMM, N = 8192
Batch-1 GEMV
Decode attention
Prefill of 2,048 tokens
The H200 ridge point is about 206 FLOP/B: below it a kernel is bandwidth-bound, above it compute-bound.
∞SOURCES
Sources and notes
All content comes from the ten public slide decks in this folder, 1_intro to 10_llm_serving_basics_2a, Harvard CS2680, Juncheng Yang, course site cs2680.com.
Third-party data cited in the slides keeps its source: ReAct (arXiv 2210.03629), Liu et al., “Lost in the Middle” (TACL 2024), Chroma’s Context Rot, Breunig, “How Contexts Fail” (2025), Lewis, “Same Model, Different Harness” (arXiv 2608.26218), Anthropic, “Building a C compiler”, Codex and OpenAI internal usage reports (June and September 2026), Meta’s Llama 3 paper, NVIDIA whitepapers and datasheets, and each benchmark’s original paper.
Left out: course overview, grading and policy, assignments and deadlines, computing resources, in-class exercises and “next class” slides. The two assignment-sharing PDFs (assignment3_sharing, assignment3_feedback) are password-protected and were not read.
Charts are redrawn from the values in the slides. The attention and FFN totals for Llama 3 70B are the per-layer figures in the slides times 80; every other value is as given in the slides.