What this course is and is not, the stack we spend the semester walking down, and the back-of-the-envelope calculation showing that for any model with real usage the lifetime bill is dominated by serving rather than training.

A transformer block as a small pile of matrix multiplies: parameter counting, where the FLOPs go, and the four consumers of training memory. Then the training loop and where its factor of three comes from, and the worked budget that puts the reference 7B’s persistent state at 107.8 GB against an 80 GB card — which is why distributed training is a requirement rather than an optimization. Then the same model forwards only: prefill against decode, the roofline, and the collision that organises the rest of the semester — the hardware wants a decode batch of 295 and the KV cache permits 29.

What an agent is, minimally. Where the tokens actually go across a multi-turn session, why context is the scarce resource, and how to tell a model failure from a harness, tool, or specification failure. Assignment 1 is in flight, due Sep 20.

Writing the loop yourself rather than importing one: tool interfaces as API design, retries and error recovery, stopping conditions, and context management as a cache policy. Assignment 2 goes out.

The distance between a loop that runs and an agent that works: the specification you ship and the two lines of it your own code reads, the verifier placed inside the loop and what its output costs, and the frozen task set that is the only instrument able to say whether a change helped. Then diagnosis, instrumentation, and when one context is not the design.

Whether to hand a task to an agent at all — your time is the specification plus the checking divided by the success rate, so verification cost decides, not model quality. Then the verifier that turns a 36% task into an 82% one, the four cost knobs you hold from outside the API, and the arithmetic for when fanning work across several agents beats running it in one context.

Written as the opening of Part III, now optional. An agent’s usefulness is governed by the cost of checking its work, and systems tasks are unusually blessed with cheap, legible, non-gameable checkers built for other reasons. SWE-agent’s finding that the interface beat the model, why fast_p is two-part, and five questions to ask every paper in this area.

The last link in the chain, and the one the course spends the semester pricing: an agent pointed at research itself. What the automated-research benchmarks actually score, why a rising score with an unchecked claim underneath is the failure mode both MLGym and AI Scientist-v2 are built against, and what a semester’s worth of serving arithmetic says about the budget line of running one of these programs.

How a GPU trades everything a CPU uses to make one thread finish sooner for the ability to keep thousands in flight: the memory hierarchy, occupancy and latency hiding, coalescing, tensor cores, and Triton as a language in which a kernel is a readable object. The semester’s one GPU programming meeting, and the machine every later number is measured against.

Attention as normally written moves a matrix nobody wants through HBM twice, which pins it at 64 FLOP/byte and 22% of an H100. Keeping the score matrix on chip makes intensity S/2 and crosses the ridge point at 590 tokens — so FlashAttention’s memory win is categorical (68.7 GB that does not fit becomes 33.5 MB) and its speed win obeys Amdahl. Then why none of it rescues decode.

A serving engine is mostly an allocator, and the naive one reserves KV to the maximum sequence length: 15% memory utilization and 29 concurrent sequences where the bytes would have held 200. Paging the cache fixes it — 29 to 199 — and the same block table turns out to make sharing, migration, and preemption possible. After it, the SLO binds before the memory does.

A static batch runs until its longest member finishes, so a batch of thirty-two is a batch of eight. Re-forming the batch every iteration recovers 4.05× from bookkeeping alone, and it works only because the cache is paged. Then selective batching, head-of-line blocking priced in milliseconds, and the length prediction nobody has.

Prefill is compute-bound and latency-tolerant; decode is bandwidth-bound and latency-critical; continuous batching put them in the same queue. Two answers: chunk the prefill until no step is dominated by it, or split the phases onto separate pools and ship the KV cache between them. The pool ratio a chat workload wants is 1.3 to 1; an agent workload wants 12 to 1.

One card became many, and an LLM replica has warm state — so the router is a cache-placement policy that also has to respect queueing and capacity. Live migration of a running request costs 1.3 ms over NVLink against 129 ms to kill and recompute, and the affinity rule is an inequality: chase the prefix when the extra queueing is worth less than the prefill it saves.

Every number in Part II is proportional to 512 KiB per token, and there are four ways to attack it. Fewer heads, fewer bits, fewer tokens — composing to 25.6× and taking concurrency from 29 to 744. It is also where Part II crosses the line from exact optimizations to approximate ones, which need a different kind of evidence.

The two families that do not shrink the cache: read fewer of the tokens you kept, which buys TPOT and no concurrency at all, and move the bytes to another tier, which buys capacity and costs TPOT. One question sorts them — how many sequences fit — and one inequality, on its fifth appearance, prices every tier against recomputing.

Causal attention makes a prefix’s KV cache a pure function of that prefix, so the 92% of prompt tokens an agent re-sends becomes recoverable machine time. RadixAttention’s tree, cache-aware scheduling, what a hit rate is worth in prefill FLOPs and TTFT, and the eviction policy the design space actually turns on.

How GPTQ and AWQ reach 4 bits with no retraining, by two opposite philosophies — compensate every rounding error, or protect the weights that matter. Then the half the accuracy tables leave out: why a 4-bit checkpoint can serve slower than an 8-bit one when dequantization lands in the GEMM’s main loop.

The decode floor says one pass over 13.5 GB yields one token, and 99.66% of the arithmetic is idle. A cheap draft proposes several tokens and the target checks them all in one pass, with a rejection rule that preserves the output distribution exactly. The catch: it spends spare FLOPs, so at a large enough batch it becomes a net loss.

What becomes possible when the application tells the serving system its structure instead of handing it finished strings: Parrot’s Semantic Variables and the explicit request DAG, the three things they buy, and the same 20-step task scheduled with and without that information.

The second half of the Nov 18 meeting. What a scheduler can do when the application declares nothing — because most clients never will. Autellix recovers most of shortest-job-first without clairvoyance by accounting service per program rather than per request, which also exposes a 67× fairness bug hiding inside a neutral-looking policy.

Half an agent session’s wall clock is spent generating nothing, and the session does not become free while it waits — it holds its KV cache, its scheduler slot, and a sandbox nobody budgets for. The stall priced in three currencies, the two break-evens deciding keep, swap, or discard, and a throughput expression that stops depending on how long the tools take.

Where a stalled session’s cache should live when the answer is “not here”. The fetch-versus-recompute line at session scale, the tier table, and what the client’s context policy does to the server’s cache — including the retention question nobody sets deliberately.

Multi-agent workloads as a serving problem: four shapes, context isolation priced, why fan-out is the best case a prefix cache ever sees, and what a barrier does to a scheduler that cannot anticipate it. Then multi-agent systems used for performance optimization, and whether a fleet beats a good ruler.