Lecture 2 said batching is the only lever that raises decode's arithmetic intensity, and Lecture 10 made a large batch affordable by fixing the allocator. Both were talking about the batch as a number. Today it becomes a set that changes every step, and the reason is a defect the number hides: a batch of thirty-two sequences whose output lengths differ runs until the longest one finishes, so with realistic lengths its average occupancy is under eight. Three quarters of the concurrency you paid for is padding. The fix is to stop treating the batch as a unit of admission and start treating the iteration as the unit of scheduling — which is Orca's contribution, and which is only implementable on top of Monday's block table. By the end you should be able to compute the tail waste of a static batch from a length distribution, explain why the linear layers batch and attention does not, price head-of-line blocking in milliseconds, and say what a perfect length oracle would and would not buy.

Where this sits

Part II has spent three meetings making a single forward pass efficient and affordable. Lecture 2 established that decode is bandwidth-bound and that only batching raises its intensity, with a ridge point demanding B ≈ 295. Lecture 10 found that a reserve-to-maximum allocator was throwing away 85% of the KV budget and that paging takes concurrency from 29 to 199 — and ended on the inversion that matters: after paging, the SLO binds before the memory does. Wednesday and Monday last week went below that to the kernel, and left it a fixed cost.

So the machine can now hold a large batch. Today asks whether it actually runs one, and the answer is no — for a reason that has nothing to do with memory or kernels and everything to do with the fact that requests arrive and finish at different times. This is the first lecture in Part II whose subject is time rather than space, and it is where the engine stops being an allocator and becomes a scheduler. Wednesday (disaggregation, Oct 7) takes the one problem today cannot solve.

Instructor notes — Timing plan

75-minute class (Mon/Wed 11:15am–12:30pm, SEC 2.118). Instructor-led, one paper, and one number that has to land in the first fifteen minutes or nothing after it means anything.

Time	Segment	Notes
0–6	Framing: the batch is a set, not a number	Re-board Lecture 10's B = 164, then ask what the 164 sequences are doing.
6–20	§11.1 Tail waste	Protected — never cut. Derive effective B = 7.9 from 32. The room must feel the padding.
20–34	§11.2 Iteration-level scheduling	Protected. The timeline picture, then the dependence on Monday's block table.
34–46	§11.3 Selective batching	The part they will skip in the paper. Make them tell you which operator cannot batch.
46–58	§11.4 Objective and head-of-line	The 436 ms vs 169 ms derivation live. Then who pays.
58–65	§11.5 Admission and preemption	Short — it is Lecture 10 §10.5 at higher frequency.
65–71	§11.6 What it buys	1,102 → 4,469 → 8,159 tok/s. Three numbers, one table.
71–75	§11.7–§11.8 Length prediction, and Wednesday	One sentence each. End on the 6.7× spike.

Reading-only, not scheduled: §11.7 in full, and the fairness discussion at the end of §11.4.

If running long: compress §11.5 to one sentence (preemption is now a per-iteration decision) and drop §11.6's third column. Never cut §11.1 or §11.2 — the tail-waste arithmetic and the timeline are the lecture, and §11.3 is what makes §11.2 credible rather than magical.

Learning objectives

By the end of this class you should be able to:

1. Compute the effective batch size of a static batch from a distribution of output lengths, and convert it into lost throughput.
2. Explain iteration-level scheduling, and state the property of the memory allocator it depends on.
3. Say which operators in a transformer layer can be batched across sequences of unequal length and which cannot, and explain why in terms of the shapes.
4. Price head-of-line blocking for a given arrival mix, and compute the mean-TTFT difference between FCFS and shortest-first ordering.
5. Explain why shortest-job-first is not implementable in an LLM engine, and what would have to be true for it to be.
6. Decide, per iteration, whether to admit, run, or preempt, given a memory watermark and the swap-versus-recompute threshold.
7. Compute the throughput of static, continuous, and SLO-capped continuous batching for a stated workload, and say which constraint binds in each case.
8. Name the problem continuous batching does not solve, and the two families of answer to it.

11.1 A batch of thirty-two is a batch of eight

Start with how batching worked before 2022, because it is how batching works everywhere else and the failure is invisible until you look for it. A server collects requests until it has B of them, runs them together to completion, then collects the next B. Call this request-level or static batching. It is the right design for image classification, where every input takes the same number of passes, and it is catastrophic for generation, where they do not.

The reason is one line. A static batch occupies the device until its longest member finishes, but it does useful work only for the members that are still generating. Every step after a sequence emits its end-of-sequence token, that sequence's slot is padding: it still costs a slot in the batch dimension, it still contributes its KV cache to the step's byte count, and it produces nothing.

That ratio is small, and how small depends on the tail. Output lengths in real traffic are heavily skewed — most replies are short, a few are very long — so take the exponential distribution as an honest first model, with mean 128 tokens.

Read that carefully, because it is worse than a 4× throughput loss. The batch dimension is what buys arithmetic intensity (Lecture 2 §2.14: intensity ≈ B), so a nominal B = 32 that averages 7.9 is also running at a quarter of the intensity you sized the deployment for. And the padding is not free in memory either: those finished sequences hold their KV blocks until the whole batch retires, so a static engine's memory high-water mark is set by the batch's longest member as well.

Two properties of that 24.7% deserve to be stated, because they mean the problem cannot be tuned away. First, it gets worse as B grows: H_n grows like ln n, so efficiency ≈ 1/(ln B + γ) falls monotonically. B = 8 gives 36.8%, B = 32 gives 24.7%, B = 128 gives 17.0%. The larger the batch you build to satisfy the ridge point, the smaller the fraction of it that is real. Second, it is driven by the tail, not the mean, so tightening the average reply length does not help; truncating the maximum does, which is why max-token caps are load-bearing in production and why Lecture 4 §4.2 put output caps in the tool rather than in the prompt.

The obvious patches all fail in instructive ways. Bucketing requests by predicted length needs a prediction you do not have (§11.7). Using a smaller batch reduces the waste fraction but abandons the intensity you were batching for. Padding to a fixed length is what we are already doing. The actual fix is to stop making the batch an admission unit.

Instructor notes

Minutes: 14. Protected. Board: Draw 32 horizontal bars of wildly unequal length, left-aligned. Shade the useful part of each. Then draw the vertical line at the longest. The white space to the left of that line is the lecture — point at it and say nothing for a beat. Ask the room: "What is the batch doing at step 400?" Answer: running one sequence and 31 slots of padding, at full memory cost. Expect confusion: Students assume finished sequences are dropped. In static batching they cannot be — the tensor's batch dimension is fixed for the batch's lifetime. That is the whole point. Common wrong answer: "Sort the requests by length." You do not know the output length at arrival; §11.7 is what that would buy if you did.

11.2 Iteration-level scheduling: re-form the batch every forward pass

Orca's answer is to move the scheduling decision from the request boundary to the iteration boundary. The engine's loop becomes: choose the set of sequences to run in the next forward pass, run exactly one pass, append each sequence's emitted token, retire whoever finished, admit whoever fits, repeat. A sequence joins the batch at the next iteration boundary after it arrives and leaves at the end of the iteration in which it finishes. Nothing waits for anyone else.

The effect on §11.1's arithmetic is total: there is no longer any such thing as the batch's longest member, because there is no longer any such thing as the batch. A slot vacated at step 137 is occupied by a new arrival at step 138. Occupancy stops being mean ÷ max and becomes simply "as many sequences as memory and the SLO allow", which is the 164 of Lecture 10 §10.6. Effective batch size goes from 7.9 to whatever you set the cap to, and the entire 4× of §11.1 is recovered by bookkeeping.

This is called continuous batching in most engines' documentation, and the name is better than "iteration-level scheduling" for conveying what a user sees: a request's latency stops depending on who it happened to be batched with.

Now the part the paper is careful about and summaries are not. Iteration-level scheduling is only implementable if adding and removing a sequence is cheap. In a static engine, the KV cache for a batch is one contiguous tensor whose batch dimension is fixed at allocation; inserting a new sequence means reallocating and copying, which at Lecture 10's numbers is gigabytes of movement while 163 other sequences wait on the step. Monday's block table is exactly what removes that cost: a sequence's cache is a list of block references, so joining the batch is appending an entry to a table and leaving it is dropping one. Orca predates PagedAttention and manages KV at a coarser granularity, which is why reading the two together in that order is the right way round — Orca establishes why you want to reshape the batch every step, and vLLM makes the reshaping free.

One honest caveat about what continuous batching does to latency, because it is the mirror of Lecture 10 §10.6's finding. Because TPOT is a property of the batch, keeping the batch full at all times means every sequence sees the full batch's step cost for its whole life, rather than enjoying the cheap steps a draining static batch provides at the end. Continuous batching strictly improves throughput and slot utilization; it improves mean latency by removing queueing, and it can make a lucky request's TPOT slightly worse. Report both, as always.

Instructor notes

Minutes: 14. Protected. Board: The six-request timeline, twice — static above, continuous below, same time axis. The picture does the work; do not derive first. Ask the room: "What has to be true about memory for the bottom picture to be implementable?" Push to: adding a sequence must not require moving anyone else's cache. Then name Monday's block table. Expect confusion: Students think continuous batching is a bigger batch. It is the same cap, kept full. Say: "We did not raise the ceiling; we stopped leaving the room half empty."

11.3 Selective batching: why the batch is not one shape

Here is the obstacle that makes §11.2 non-trivial, and the reason Orca needed a second idea. If the sequences in an iteration have different context lengths — and under continuous batching they always do, since they arrived at different times — then the tensors flowing through the layer are ragged, and not every operator can consume a ragged batch.

Sort the layer's operators by whether they care.

The linear layers do not care. A projection multiplies each token's d_model-vector by a shared weight matrix. Whether the step holds 200 tokens from one sequence or one token from each of 200 sequences is irrelevant: flatten them into a single (T × d_model) activation matrix, where T is the total token count in the step, and issue one GEMM against the same weights. This is exactly the property that makes batching work at all — the weights are read once and amortized over T tokens — and it holds for the QKV projection, the output projection, and all three MLP matrices, which together are Lecture 2 §2.4's 202,375,168 parameters per layer, the whole weight bill.

Attention cares, absolutely. Each sequence attends over its own KV cache, of its own length. There is no shared operand to amortize and no single dense shape to form: sequence i needs a (1 × d) query against an (S_i × d) key matrix, and the S_i differ. Concatenating them into one dense GEMM would require padding to max S_i, which reintroduces §11.1's waste inside every single step.

So Orca batches selectively: it flattens the token dimension for every linear operator and handles attention per sequence, splitting the flattened tensor before attention and re-merging after. That is the whole mechanism, and it has three consequences worth stating.

First, it explains why the attention kernel is the hard part of every serving stack, and closes the loop with Wednesday: Lecture 9 §9.8 said decode attention is 1 FLOP/byte with no temporary to fuse, and now we can see it is also the only operator that cannot be batched into a big GEMM. Those two facts are the same fact. The modern answer is a kernel that takes a ragged batch and a block table and does the gather itself — which is FlashInfer's job, and why Lecture 10 §10.3 insisted PagedAttention is an allocator and a kernel.

Second, it tells you what a step's cost actually is, and it is not "B times something". Bytes per step = 13.5 GB of weights (read once, regardless of T) + Σ over sequences of their KV. Lecture 2 §2.15's warning falls out immediately: the weight term is fixed and the cache term grows with the batch, so at B = 164 with 640-token MHA sequences the step moves 13.5 + 53.8 = 67.3 GB and only 20% of it is weights. Batching amortizes the weights and does nothing whatsoever for the cache.

Third, it is why a step can mix prefill and decode tokens at all. A prefilling sequence contributes many tokens to T and a decoding one contributes a single token; the linear layers cannot tell them apart. That flexibility is what Wednesday's chunked prefill exploits — and §11.8 is why it is needed.

Instructor notes

Minutes: 12. Board: Two boxes. "LINEAR: flatten to (T × d_model), one GEMM, shared weights." "ATTENTION: per-sequence, own cache, own length." Then: "these are the same fact as Lecture 9 §9.8." Ask the room: "Which operator stops you batching sequences of different lengths?" Make them find attention. Then: "Why exactly?" Own cache, own length, no shared operand. Expect confusion: "Just pad the caches to the same length." Price it out loud: padding to max S_i inside every step is §11.1's waste, per step, forever.

11.4 What the scheduler is actually optimizing, and who pays

Continuous batching decides when a sequence may run. It does not decide which sequences to prefer when more want to run than fit. That is the scheduling policy, and the classical results apply with one classical obstruction.

The queueing theory is standard: shortest-job-first minimizes mean waiting time, and first-come first-served maximizes fairness and is the only policy immune to starvation. The LLM-specific twist is that jobs differ in cost by three orders of magnitude — a 200-token reply against a 32K-token prefill — so the gap between the two policies is enormous, and you do not know a job's length at arrival. Prefill length you know; output length you do not.

Price the gap concretely, because "head-of-line blocking" is easy to nod along to and hard to feel.

That is the whole tension, and it is worth being explicit about the ethics of it rather than only the arithmetic. Mean latency is a metric that can be improved by systematically disadvantaging a minority of requests, so a scheduler tuned on the mean will discover that strategy on its own. Two guards are standard. Aging raises a waiting request's priority with its wait time, converting starvation into bounded unfairness. Reserving capacity — a fraction of slots that only long requests may use — bounds the damage directly. Both cost mean latency to buy tail latency, which is the same trade Lecture 2 §2.16 insisted on reporting as percentiles rather than means.

There is also a second dimension of fairness that classical queueing does not have, and it returns with force on Nov 18: an agent is not one request but hundreds of dependent ones, so any per-request policy — FCFS included — silently favours whoever issues more requests. A twenty-step agent competing with twenty single-turn chats gets one twentieth of the service a fair share would give it, and no per-request scheduler can see the difference. Hold that; it is Autellix's whole argument.

Instructor notes

Minutes: 12. Board: Two rows of boxes, one long box among short ones, drawn in each order. Then the 436 / 169 pair, then "the long request: 294 → 578". Ask the room: "Which policy would you ship?" Force the follow-up: "for whom?" A chat product and a batch-summarization product answer differently. Expect confusion: Students treat SJF as strictly better. Make them name the victim. Common wrong answer: "Use the prompt length as the job length." It predicts prefill cost well and total cost badly — a 20-token prompt can generate 4,000 tokens. §11.7.

11.5 Admission and preemption, now decided every iteration

Lecture 10 §10.5 established the engine's three moves when memory runs out — refuse, preempt, or crash — and derived the preemption arithmetic: swapping a 4K MHA sequence costs 67 ms round trip against 129 ms to recompute its prefill, so swap wins 1.9× (7.7× under GQA-8, because shrinking the cache does not shrink prefill). None of that changes today. What changes is the frequency: the decision is now made at every iteration boundary rather than at every batch boundary, which is hundreds of times more often.

That raises the stakes on the watermark. A continuous scheduler is admitting new sequences whenever a slot frees, so it is continually walking up to the memory limit; without a reserve of free blocks it will admit a sequence and then discover, two steps later, that the resident set has collectively grown past capacity. Engines therefore keep a watermark of free blocks and stop admitting below it, which is the cheapest of the three moves because it hurts only requests that have not yet started — and it is why Lecture 10's finding that the SLO binds before the memory does is such a relief: a well-configured engine spends its life nowhere near the preemption threshold.

Two refinements are worth naming. Preempt the newest, not the oldest. The victim should be the sequence with the least accumulated work, since preemption discards or displaces its cache and the cost is proportional to what it has built up; this is also the choice that best preserves the completion of requests already close to done. Recompute is not always the loser. At Lecture 10's break-even bandwidth — 33.3 GB/s for MHA, 8.3 GB/s for GQA-8 — a slow interconnect flips the decision, and Exercise 4 of that lecture showed the crossover moves with sequence length because the cache is linear in S while prefill is quadratic. A single global policy is therefore wrong for a mixed workload, and the per-sequence decision is computable from its length alone.

Instructor notes

Minutes: 7. Board: "refuse / preempt / crash" from Monday, then "× hundreds of times more often". Ask the room: "Which sequence should you preempt?" Push to least-accumulated-work, and make them say why: the cost of preemption scales with what you throw away. Expect confusion: Preemption is thought to lose the request. It loses its cache; the tokens are the state (Lecture 4 §4.5), so recovery is replay.

11.6 What it buys, in this semester's numbers

Assemble the three regimes on one workload, so the gain is attributable rather than asserted. Reference 7B, MHA, one H100, 640-token mean occupancy (0.328 GB of KV per sequence), exponential output lengths with mean 128.

	Static B = 32	Continuous, cap 32	Continuous, SLO-capped
Effective concurrency	7.9	32	164
TPOT	7.16 ms	7.16 ms	20.1 ms
Throughput	1,102 tok/s	4,469 tok/s	8,159 tok/s
Binding constraint	the batch's tail	the configured cap	the SLO
$/Mtok at rate R	0.252·R	0.062·R	0.034·R

Three things to take from the table. The 4.05× is free in the strict sense that no hardware, model, or kernel changed — it is the same steps in a different order. The further 1.83× is not free: it spends 13 ms of TPOT to buy it, which is exactly the frontier Lecture 2 §2.16 said the required paper reports instead of a number. And the binding constraint has now moved twice in two lectures — from the allocator (Monday) to the schedule (today) to the SLO — which is the honest shape of systems work and the reason the semester's remaining lectures attack bytes and structure rather than mechanism.

Instructor notes

Minutes: 6. Board: The three-column table, filled left to right, saying the binding constraint aloud each time. Circle 1,102 → 4,469 → 8,159. Ask the room: "Which of those two multipliers would you take first?" The free one. Then: "what did the second one cost?" 13 ms of TPOT.

11.7 The input the scheduler does not have

Reading-only; not scheduled in class.

Every hard decision above — shortest-first, exact memory reservation, bucketing, admission without a watermark — needs one number nobody has: how many tokens will this request emit? It is worth being precise about what a perfect oracle would buy, because it bounds the whole research area.

With an exact output length known at arrival: shortest-job-first becomes implementable, so §11.4's 2.6× mean-TTFT gain is available with aging as the only guard. Memory can be reserved exactly rather than watermarked, so preemption disappears and the engine can run at the true memory ceiling instead of below a safety margin. And a scheduler could pack sequences whose completions are staggered, smoothing the step cost.

With a wrong prediction the failures are asymmetric, which is the part that matters for design. Under-predicting is dangerous: the engine has promised memory it does not have, so it must preempt — paying 67 ms or 129 ms per victim (§11.5) — and it has scheduled a long job as though it were short, so the mean-latency gain it was chasing evaporates. Over-predicting is merely wasteful: capacity is reserved and unused, so the engine refuses admissions it could have accepted, and you are back toward Lecture 10 §10.2's reserve-to-maximum regime. The sensible engineering response is therefore not a better point estimate but a conservative one, plus a mechanism that survives being wrong — which is what a watermark plus cheap preemption is.

The literature reflects this. Predictors trained to estimate response length work well enough on aggregate to help with bucketing and badly enough on individual requests that no production engine bets memory on them. It is a good example of a general rule worth carrying into Part III: a prediction is only useful to a system that can afford to be wrong about it, and the cost of being wrong is a systems question, not a modelling one.

11.8 What continuous batching does not fix

Today recovered 4× from the schedule and moved the binding constraint to the SLO. One problem is strictly worse than before, and it is Wednesday's subject.

Continuous batching mixes whoever is ready into the same forward pass. §11.3 showed the linear layers cannot tell a prefilling sequence from a decoding one, which is what makes the mixing possible — and it is also what makes it dangerous, because the cost of the two is wildly different. A 4,096-token prefill contributes 4,096 tokens to T and costs 64.0 TFLOP ≈ 129 ms, while a decode step for 164 sequences costs 20.1 ms of memory time. Put one of those prefills into the batch and every resident sequence's TPOT goes from 20.1 ms to about 149 ms — Lecture 2 §2.16's 6.7× spike, inflicted on 164 strangers because one request had a long prompt.

Worse, continuous batching increases the frequency of this, because it admits new arrivals as soon as slots free rather than at batch boundaries. The very mechanism that removed queueing delay has made prefill interference a per-second event rather than a per-batch one. That is the honest cost of today's win, and it has exactly two families of answer: chunk the prefill so no single step is dominated by it, or disaggregate the phases onto separate pools so they never share a step. Wednesday is both.

Discussion seeds

1. The tail is the enemy. §11.1's efficiency is 1/(ln B + γ) for exponential lengths. Redo it for a bimodal workload — 90% of replies at 64 tokens, 10% at 2,048 — and say which of the two modes you would attack with a product decision rather than a systems one.
2. Selective batching's cost. Orca splits the flattened tensor before attention and re-merges after. What does that split cost, and at what batch size would the bookkeeping start to matter against a 20.1 ms step?
3. Who should the scheduler serve? §11.4 improved mean TTFT 2.6× by nearly doubling one request's latency. Write the SLO that would forbid that trade, and say what throughput it costs.
4. Agents break per-request fairness. A 20-step agent competes with 20 single-turn chats under FCFS. Compute each party's share of service, then design the smallest change to the scheduler that fixes it. (Nov 18 has an answer; find yours first.)
5. The prediction you would actually deploy. Given §11.7's asymmetry, design a length predictor's output — not its model. What should it return so that a scheduler can use it safely?
6. What is left. NanoFlow argues a well-batched engine still under-uses the device because compute, memory, and network idle in turn. Using §11.6's SLO-capped column, estimate how much of the H100 is idle at B = 164, and say which resource you would overlap first.

Key takeaways

Numbers worth memorizing

Quantity	Value	Source
Static-batch efficiency	mean ÷ max output length = 1/(ln B + γ)	§11.1
B = 32, exponential mean 128	E[max] = 128·H₃₂ = 519 steps, efficiency 24.7%	§11.1
Effective batch, nominal 32	7.9	32 × 0.247
Efficiency at B = 8 / 32 / 128	36.8% / 24.7% / 17.0%	worsens with B
Continuous-batching gain	4.05×, from scheduling alone	32 ÷ 7.9
512-token prefill	7.04 TFLOP → 14.2 ms	50% of peak
8,192-token prefill	145.6 TFLOP → 294 ms	Lecture 13
FCFS vs shortest-first, mean TTFT	436 ms vs 169 ms (2.6×)	§11.4
Price paid by the long request	294 → 578 ms	§11.4
Throughput: static / continuous / SLO-capped	1,102 / 4,469 / 8,159 tok/s	§11.6
Bytes per step	13.5 GB weights + B × KV; 20% weights at B = 164	§11.3
Prefill interference after continuous batching	20.1 ms → ~149 ms = 6.7×, more often	§11.8

Self-check

Exercises

1. A bimodal workload. Replace §11.1's exponential lengths with: 90% of replies at 64 tokens, 10% at 2,048. For B = 32, compute the expected maximum, the mean, the static efficiency, and the effective batch. Then say which single change — systems or product — recovers the most. Solution sketch: Mean = 0.9·64 + 0.1·2,048 = 262.4. P(no long reply in 32) = 0.9³² = 0.0342, so with 96.6% probability the max is 2,048; E[max] ≈ 0.966·2,048 + 0.034·64 ≈ 1,981. Efficiency = 262.4 ÷ 1,981 = 13.2%, effective batch 4.2 — much worse than the exponential case's 24.7%, because a single 2,048-token reply holds all 32 slots open. The best systems change is continuous batching (recovers the full 7.6×). The best product change is a max-token cap: capping at 512 tokens leaves 90% of traffic untouched, drops E[max] to 512, and lifts static efficiency to (0.9·64 + 0.1·512)/512 = 21.3%. A single cap outperforms every scheduling trick available to a static engine, which is why Lecture 4 §4.2 put the cap in the tool.
2. Where selective batching's bookkeeping starts to matter. A step at B = 164 costs 20.1 ms. Suppose splitting and re-merging the flattened activation around attention costs one extra pass over the step's activations. Compute that cost at d_model = 4096, bf16, and give the batch size at which it reaches 1% of the step. Solution sketch: The step holds T = 164 tokens in decode (one per sequence). One extra read+write pass over the activations is 2 · T · 4096 · 2 B = 16,384·T bytes = 2.69 MB at T = 164, which at 3,350 GB/s is 0.80 µs — 0.004% of a 20.1 ms step. It reaches 1% when 16,384·T ÷ 3.35e12 = 0.01 · step, and since the step itself grows with T, the ratio actually falls: the split cost is linear in T while the step's dominant term (13.5 GB of weights) is constant. Selective batching's overhead is negligible by construction, which is why Orca could afford it — and the same argument fails for a prefill-heavy step, where T is thousands rather than hundreds and the activations are the dominant traffic (Lecture 2 §2.7's 3.69 MB/token).
3. Aging, priced. Implement §11.4's aging guard as: a request's priority is its prefill cost minus k × milliseconds waited. For the 21-request mix, find the k that caps the long request's TTFT at 400 ms, and compute the resulting mean TTFT. Solution sketch: Under shortest-first the long request waits 284 ms then prefills for 294. To cap TTFT at 400 ms it must start by 106 ms, i.e. after at most 106/14.2 = 7 short requests. Its effective priority must beat a short one's after 106 ms of waiting: 145.6 − k·106 ≤ 7.04, so k ≥ 1.31 TFLOP/ms. Then 7 shorts go first (mean TTFT 4·14.2 = 56.8 ms), the long one runs 99.4–393.4 ms, and the remaining 13 shorts finish by 578 ms with mean TTFT 393.4 + 7·14.2 = 492.8 ms. Overall mean = (7·56.8 + 393.4 + 13·492.8)/21 = 324 ms — worse than pure SJF's 169 ms, better than FCFS's 436 ms, with the tail capped. Aging buys a bounded tail for 1.9× the mean latency, and that exchange rate is the thing to report, not the policy name.
4. Does the ridge point ever become reachable? Lecture 2 wants B = 295. Using §11.6's step-cost model, find the per-sequence KV footprint at which a 20 ms TPOT SLO permits B = 295, and state what configuration achieves it. Solution sketch: 20 ms × 3,350 GB/s = 67.0 GB per step; subtract 13.5 GB of weights, leaving 53.5 GB for 295 sequences = 0.181 GB each. At 512 KiB/token MHA that is 354 tokens — useless. At 128 KiB/token GQA-8 it is 1,416 tokens — a plausible chat context. So the ridge point is reachable at GQA-8 with contexts under ~1.4K, and unreachable for MHA or for long context at this SLO. Loosen the SLO to 30 ms and the allowance becomes (100.5 − 13.5)/295 = 0.295 GB = 2,360 GQA-8 tokens. The ridge point is not a fantasy; it is a joint constraint on architecture, context length, and SLO — which is the precise sense in which grouped-query attention was a serving decision.
5. Attribute a measured gain. You replace a static engine (B = 32) with a continuous one and measure 3.1× throughput, not the predicted 4.05×. Give three candidate explanations, each with the measurement that would confirm it, and say which you would check first. Solution sketch: (a) Output lengths are less skewed than exponential — the predicted gain is 32/effective-B, so measure the actual length distribution and recompute mean ÷ E[max]; a bimodal or truncated distribution gives a smaller ratio. (b) The engine is not actually keeping the batch full — measure occupancy per iteration and look for admission stalls at the watermark, which would show as free slots coexisting with a non-empty queue. (c) Prefill interference — §11.8: continuous batching admits prefills more often, so measure the fraction of steps containing prefill tokens and the TPOT distribution; a bimodal TPOT confirms it. Check (a) first, because it is one query against a log and it recalibrates the prediction rather than blaming the system — and because if the distribution is the answer, (b) and (c) are wild geese. The general habit is to validate the model's inputs before debugging the system, which is Lecture 5 §5.5's diagnose-in-order rule applied to your own arithmetic.

Reading guide

Required — Orca. Read §2–§3 quickly for the setup; the paper's framing of the problem is §11.1 and you already have it. Then §4, twice. The first pass is for iteration-level scheduling, and the thing to extract is the invariant: what must be true at an iteration boundary for a sequence to join or leave. The second pass is for selective batching, which is the half that summaries omit and the half that makes the idea real — do not move on until you can list, from memory, which operators batch across ragged lengths and which do not, and say why in terms of shared operands. §5's scheduler is worth reading for the mechanics of the request pool and the max-token budget; skim §6, and treat its multipliers as measurements on its workloads rather than as constants, since §11.1 shows the gain is a pure function of the length distribution. Hold this question: iteration-level scheduling requires cheap insertion into a running batch — what does Orca assume about KV allocation, how does that differ from Monday's block table, and what does the difference cost them?

Optional — vLLM / PagedAttention. Monday's required paper, re-read for one paragraph: the block table is what makes today's scheduling free. If you read only one thing, read §4.4 on preemption against §11.5. Question: which of Orca's limitations does paging remove outright, and which survive?

Optional — LoongServe. What today's ideas become when sequences are long enough that a single one does not fit comfortably on one device, so the elastic quantity is the parallelism degree rather than the batch. Read the elastic-sequence-parallelism design; skip the implementation. Question: §11.3 said attention is the operator that cannot be batched — what does that imply when one sequence's attention is itself split across GPUs?

Optional — NanoFlow. The natural sequel: even a perfectly batched engine leaves compute, memory, and network idle in turn, and the proposal is to overlap them within a device. Read the pipelining argument and the resource-utilization measurements. Question: using §11.6's SLO-capped column, what fraction of the H100 is idle at B = 164, and does NanoFlow's answer or Wednesday's disaggregation recover more of it?

Looking ahead

Wednesday (batching and scheduling II — disaggregation, Oct 7) is a student-led paper discussion, and it takes the one problem today made worse. Continuous batching mixes a 129 ms prefill into a 20.1 ms decode step, and does so more often than static batching ever did, so every resident sequence pays Lecture 2's 6.7× TPOT spike for a stranger's long prompt. There are two answers: chunk the prefill until no step is dominated by it (Sarathi-Serve, the required reading), or disaggregate the phases onto separate pools so they never share a step (DistServe, Splitwise). Come with a view on which you would deploy on one card and which on a cluster — and read the paper discussion guide if you are presenting.

Assignment 3 (make the agent smarter, faster, cheaper) is due Oct 25, 11:59pm, and Assignment 4 (serve your own agent) goes out the same day, due Nov 10. A3 is still outside the API, so today's levers are not yours yet; A4 is where they become yours, and max_num_seqs — the cap in §11.6's second column — is one of the four options Lecture 10's reading guide asked you to map onto a quantity in these notes. You now know exactly what that quantity is.

One thing to carry out of the room. The 4.05× came from running the same steps in a different order. No byte moved faster, no FLOP got cheaper, and no model changed — the engine had simply been leaving three quarters of its concurrency idle and nobody had measured occupancy. That is the second time in three lectures that the largest available win was a measurement nobody had taken, after Lecture 10's 15% memory utilization. Ask what your own system is not measuring.