Monday recovered a factor of four by re-forming the batch every iteration, and made one thing strictly worse in the process: because a mixed step cannot tell a prefilling token from a decoding one, a 129 ms prefill now lands in a 20 ms decode step several times a second, and 164 strangers pay a 6.7× TPOT spike for one long prompt. Today is that problem and its two answers, which are genuinely different engineering positions rather than variations. Chunk the prefill until no single step is dominated by it — one card, no new hardware, a few percent of extra work — or disaggregate, running prefill and decode on separate pools that never share a step, and paying for it by shipping the KV cache across a wire. The arithmetic decides, and it decides differently for chat than for agents: the pool ratio a chat workload wants is about 1.3 to 1 and the ratio an agent workload wants is about 12 to 1. By the end you should be able to pick a chunk size from an SLO, price a KV transfer against the interference it avoids, and size two pools from an arrival rate.

Where this sits

Part II has been closing Lecture 2 §2.15's gap between the batch the hardware wants (295) and the batch the memory permits (29). Lecture 10 fixed the allocator and got to 199, then found the SLO binding first at 164. Monday found that a static batch was only 24.7% occupied and recovered 4.05× by re-forming it every iteration. Each step made the engine better at running one kind of work at a time.

Today confronts the fact that there are two kinds, they are opposites, and Monday's fix put them in the same room. Prefill is compute-bound, latency-tolerant, and arrives in bursts of thousands of tokens; decode is bandwidth-bound, latency-critical, and arrives one token at a time. Lecture 9 §9.2 and §9.8 showed just how far apart they sit — 2,048 FLOP/byte against 1 — and Lecture 2 §2.16 priced the collision. Everything after today assumes the collision is handled: Oct 14 routes requests across replicas, Oct 19 shrinks the cache, and Nov 4 reuses it. All three assume the prefill/decode question has an answer, and today is where you choose it.

Instructor notes — Discussion plan

75-minute class, student-led. This is a discussion, not a lecture: the instructor holds the arithmetic and the clock, and the room does the arguing. Two presenters minimum — one for the required paper, one taking the opposing position from DistServe or Splitwise.

Time	Segment	Who	Notes
0–8	The problem, re-boarded	instructor	20.1 ms → 149 ms, and "more often than before". No paper yet.
8–25	Presenter 1: Sarathi-Serve	student	Chunked prefill + stall-free batching. Force the token-budget question.
25–35	Derive the chunk size together	instructor	Protected. The T ≤ 683 result must be derived live, not shown.
35–52	Presenter 2: DistServe / Splitwise	student	The opposing position. Make them state what it costs, not just what it buys.
52–62	The transfer, priced	instructor	1.3 / 8.4 / 21.5 ms against 129 ms. Where the answer flips.
62–72	Pool sizing: chat vs agent	room	Protected. 1.3:1 against 12:1 is the punchline of the whole meeting.
72–75	Vote and wrap	instructor	Show of hands: one card, or a cluster? Then name Oct 14.

Questions to force, in order. (1) "Chunking makes the prefilling request slower — quantify who pays." (2) "Disaggregation moves the KV cache. What is the largest thing you would be willing to ship, and over what link?" (3) "Which of these two do you deploy if you have exactly one GPU?" (4) "Your workload is Lecture 5's agent, not a chatbot. Does your answer change?" Question 4 is the one worth the whole class.

Reading-only, not scheduled: §12.7's composition argument, and the P/D-Serve material.

If running long: cut presenter 2 to Splitwise's phase characterization alone and drop §12.5's Ethernet column. Never cut the chunk-size derivation or the pool-ratio comparison — everything else today is context for those two numbers.

Learning objectives

By the end of this class you should be able to:

1. Explain why prefill and decode interfere, in terms of arithmetic intensity rather than implementation.
2. Describe chunked prefill and stall-free batching, and say which one is the mechanism and which is the policy.
3. Derive the largest prefill chunk that fits inside a decode step without violating a stated TPOT SLO, and compute the TTFT the prefilling request pays for it.
4. Compute chunking's intrinsic overhead — the KV re-read — and say why it is small.
5. Price a KV-cache handover across a given interconnect and compare it with the interference it avoids.
6. Size a prefill pool and a decode pool from an arrival rate, prompt length, and output length, and explain why an agent workload wants a radically different ratio from a chat workload.
7. Choose between chunking and disaggregation for a stated deployment, and say how they compose.

12.1 Two workloads, one device

Restate the collision precisely, because every number today hangs off it, and state it in the vocabulary of Lecture 9 rather than as an implementation accident.

A prefill of S tokens puts S tokens through one pass over the weights. Its arithmetic intensity is in the thousands — Lecture 2 §2.14 put a 4K prefill at ≈4,100 FLOP/byte, fourteen times past the H100's ridge of 295 — so it is compute-bound, and it saturates the tensor cores. It is also latency-tolerant: nobody sees a partial prefill, so the only thing that matters is when it finishes.

A decode step puts one token per sequence through the same pass. Even fully batched it reaches ~24 FLOP/byte at GQA-8 (Lecture 2 §2.15), so it is bandwidth-bound, and it saturates HBM while leaving the tensor cores 92% idle. It is also latency-critical: every token is user-visible, and TPOT is the product.

Put both in one forward pass and the step takes as long as the harder of the two, while each phase gets in the other's way.

Note what is not the problem, because it rules out the easy fixes. It is not that prefill is expensive — 129 ms of compute for 4,096 tokens is efficient, close to the best the card can do. It is not a kernel problem: Lecture 9 already fused attention, and a faster prefill is still a prefill. And it is not fixable by refusing to mix, because a prefill-only step leaves HBM idle and a decode-only step leaves the tensor cores idle, so strict alternation wastes whichever resource is not being used. The problem is that two workloads with opposite bottlenecks are sharing one queue, which is a scheduling problem with a long history and two standard resolutions: make the units small enough that sharing is smooth, or stop sharing.

Instructor notes

Minutes: 8. Instructor holds this; no paper yet. Board: Two columns — PREFILL: compute-bound, ≈4,100 FLOP/byte, latency-tolerant. DECODE: bandwidth-bound, ≈24, latency-critical. Then "one queue" between them, and 20.1 → 149. Ask the room: "Why not just never mix them on one card?" Let them discover that alternation idles one resource or the other. That failure motivates both papers. Expect confusion: Students think this is a vLLM bug. It is a consequence of Monday's win; say so.

12.2 Chunked prefill and stall-free batching

Sarathi-Serve's answer keeps one pool and makes the units small. Two ideas, and it is worth keeping them separate because one is a mechanism and the other is a policy.

Chunked prefill (the mechanism). A prompt does not have to be prefilled in one forward pass. Split it into chunks of at most T tokens and prefill one chunk per iteration, each chunk attending over the context already written by its predecessors. This is sound for exactly the reason Lecture 9 §9.4's tiling was sound — causal attention means a token only ever needs what precedes it — and it is possible for exactly the reason Lecture 11 §11.3 gave: the linear layers consume a flat (T × d_model) activation and cannot tell where the tokens came from.

Stall-free batching (the policy). Given that mechanism, never build a step out of prefill alone. Every iteration carries all the decodes that are ready plus one prefill chunk sized to fit in the remaining budget. Decode never stalls waiting for a prefill to finish, hence the name, and prefill progresses on every step instead of monopolizing a few.

The pairing is what matters. Chunking alone would merely convert one long stall into several short ones. Stall-free batching uses the chunk as filler: the decode step is bandwidth-bound and leaves the tensor cores 92% idle, and a prefill chunk is precisely a compute-bound workload with no bandwidth appetite. The two phases are complementary at the level of the hardware's resources, and a mixed step done right uses both. That is the real insight, and it is why the right chunk size is not "small" but "exactly as large as fits in the shadow of the decode step's memory traffic".

Instructor notes

Minutes: 17, presenter-led. Board (presenter): A step drawn as a box, with decode tokens filling it and one prefill chunk tucked in beside them. Then the same picture with an unchunked prefill bursting out of the box. Force the question: "What is the token budget, and what happens at both extremes?" Budget → ∞ is today's problem; budget → 1 is 4,096 steps of overhead. Expect confusion: Students conflate the mechanism with the policy. Ask: "If I chunk but schedule chunks back-to-back with no decodes, what have I gained?" Almost nothing.

12.3 Choosing the chunk size from the SLO

This is the section to derive live, because it turns a knob into a calculation. A mixed step must do two things: move the decode step's bytes, and perform the prefill chunk's arithmetic. The first is bandwidth-bound and the second compute-bound, so to a first approximation they overlap, and the step costs the larger of the two. The chunk is therefore free — up to the point where its compute time exceeds the decode step's memory time.

Now tabulate the trade, because the prefilling request does pay — in TTFT, not in throughput.

Token budget T	Chunk compute	Step time	TPOT penalty	Chunks for 4,096	Prefill TTFT
256	7.5 ms	20.1 ms	none	16	322 ms
512	15.1 ms	20.1 ms	none	8	161 ms
683	20.1 ms	20.1 ms	none (at the edge)	6	121 ms
1,024	30.1 ms	30.1 ms	+50%	4	120 ms
2,048	60.3 ms	60.3 ms	+200%	2	121 ms
4,096 (unchunked)	120.5 ms	149 ms	+641%	1	129 ms

Read the table's shape, because it is unusually kind. Between T = 512 and T = 4,096 the prefilling request's TTFT barely moves — 161 ms against 129 ms, a 25% penalty — while the TPOT penalty inflicted on 164 other sequences goes from zero to 641%. There is a genuine sweet spot, and it is not a compromise: you give one request 32 ms of TTFT and you give 164 requests their SLO back. The right budget is the largest chunk whose compute hides inside the decode step's memory time, which is computable from the SLO and the batch, and which is exactly what max_num_batched_tokens sets in a real engine — the fourth of the four options Lecture 10's reading guide asked you to map.

Two honest caveats. The overlap assumption is optimistic: a real step does not perfectly hide compute behind memory traffic, so treat "no TPOT cost" as "small TPOT cost" and measure it. And S_ctx moves — the last chunk of a long prompt attends over far more context than the first, so the per-token cost rises through the prompt and a fixed budget makes later chunks slower. For a 32K prompt the final chunk's attention term is 16× the average, which is why long-context serving wants a shrinking budget and why this knob deserves a look before you trust one setting across a mixed workload.

Instructor notes

Minutes: 10. Protected — derive live. Board: 14.55 GFLOP/token, then divide 20.1 ms of budget by it and get 683. Then the table's two extreme rows only — T = 512 and T = 4,096 — and the words "zero against 641%". Ask the room: "Who pays, and how much?" One request, 32 ms. Make them say it. Expect confusion: "Chunking must cost throughput." It costs 1.7% of memory traffic. Show the re-read box.

12.4 The other answer: stop sharing the device

DistServe and Splitwise take the opposite position. If two workloads have opposite bottlenecks, do not schedule them together at all — give each its own pool of GPUs. Prefill instances do nothing but prefill; decode instances do nothing but decode; a request is prefilled in one pool, its KV cache is handed to the other, and it decodes there.

The case for it is that every knob then becomes independent, and there are more of them than the interference story suggests.

Each pool runs at its own optimum. A prefill pool wants large batches of tokens and cares only about tokens per second; it can run at whatever batch size saturates the tensor cores. A decode pool wants many concurrent sequences and a tight TPOT; it is sized by memory and bandwidth. Under sharing, one configuration has to serve both.

Each pool wants different parallelism. Prefill is compute-bound, so tensor parallelism buys it throughput at a communication cost it can absorb. Decode is bandwidth-bound and latency-critical, so its 64 small all-reduces per token (Lecture 2 §2.13) land squarely on the critical path — decode often prefers less parallelism than prefill. Sharing forces one choice.

Each pool can want different hardware. This is Splitwise's argument and it is the most consequential one commercially: prefill wants FLOPs and decode wants bandwidth and capacity, so the cost-optimal machine differs. Buying one SKU for both means overpaying for whichever resource the phase does not use.

The SLOs decouple. TTFT is a prefill-pool property and TPOT is a decode-pool property, so an operator can provision each against its own target instead of trading one for the other on a single frontier — which is the goodput-per-GPU framing DistServe uses.

The price is one thing, and it is the whole engineering content of the position: the KV cache has to move.

Instructor notes

Minutes: 17, presenter-led. Board (presenter): Two boxes, an arrow labelled "KV" between them, and the four independent knobs listed under the boxes. Force the question: "What does this cost?" Do not let them stop at "a transfer" — make them say how many bytes, over what link, and against what alternative. That is §12.5. Expect confusion: Disaggregation sounds obviously better because it is cleaner. It also strictly increases the number of GPUs a single request touches and adds a failure mode. Say so.

12.5 The price of separation, in bytes

The KV cache produced by prefill is the decode phase's entire input, so disaggregation must ship it. Its size is Lecture 2 §2.15's formula, and its cost is that divided by a link.

Read the table against that 129 ms and the conclusion is clear but conditional. Over NVLink the transfer is 1–4% of the interference it avoids: disaggregation is nearly free and obviously right. Over PCIe it is 6–26%: still a good trade, but now a real cost. Over Ethernet at MHA it is 86 ms against 129 ms — two thirds of the benefit consumed by the mechanism, and at that point chunking on one card is the better engineering. Grouped-query attention is what makes disaggregation comfortable, exactly as it was what made the batching ceiling comfortable in Lecture 2 and swapping comfortable in Lecture 10 §10.5: a 4× smaller cache turns a marginal transfer into an easy one, and it is the third time this semester that one pretraining decision has quietly rewritten a serving design.

Three refinements worth knowing, since they are where the papers spend their engineering. The transfer can be overlapped with the tail of prefill — ship layer i's cache while computing layer i+1 — which hides most of it behind work already being done, and is why production systems report transfer costs far below this table. The transfer can be avoided when the decode pool already holds the prefix, which is the door into Nov 4. And it can be eliminated by not moving the cache at all but making it addressable from both pools, which is what Mooncake and MemServe do and why they read as storage papers.

Instructor notes

Minutes: 10. Board: The 2×3 table, then "129 ms" underlined beside it. Point at the Ethernet/MHA cell and say "this is where the answer flips." Ask the room: "Which cell would make you choose chunking instead?" The 85.9 ms one. Expect confusion: Students assume the transfer is on the critical path in full. Layer-wise overlap hides most of it; the table is the un-overlapped worst case.

12.6 Sizing the two pools — and why agents change the answer

If you disaggregate, you must choose the ratio, and the ratio is not a constant of the system. It is a property of the workload's prompt-to-output ratio, which is exactly the quantity Part I spent five lectures measuring.

Work per request, in GPU-seconds. Prefill: P tokens × 13.48 GFLOP ÷ 494.5 TFLOP/s (weight GEMMs; the attention term adds to this and pushes the conclusion further in the same direction). Decode: G tokens × TPOT of slot time, divided by the B slots a decode GPU holds.

That contrast is the most important thing in today's meeting, and it is worth being explicit about what it means. The prefill/decode ratio is not a tuning parameter you discover empirically once; it is a direct function of how your callers use the model, and Part I showed that agent callers re-send a growing transcript on every step, which makes them prefill-dominated to an extreme degree. A cluster provisioned for chat and then pointed at agent traffic is starved of prefill capacity by an order of magnitude, and will present as mysteriously terrible TTFT with idle decode GPUs.

It also explains why Nov 4 is the lecture that matters most for agents specifically. Prefix caching attacks the 97,000 directly — 92% of it is re-sent, so a perfect cache takes the agent's prefill work from 2.64 to 0.21 GPU-seconds and the pool ratio from 12:1 to 0.95:1, which is roughly the chat shape. Prefix caching does not merely make agents cheaper; it makes them look like a different workload to your capacity planner. Hold that until Nov 4, where the same numbers reappear as a 12.6× and a 10.6×.

Instructor notes

Minutes: 10. Protected. Let the room compute the second column. Board: Two ratios, 1.32:1 and 11.97:1, then "9×". Then the Nov 4 punchline: 12:1 → 0.95:1. Ask the room: "You provisioned for chat and your users built agents. What do you observe?" Bad TTFT, idle decode GPUs. This is the most practically useful prediction in the lecture. Expect confusion: Students will want to use P = 4,850 (the average step) rather than 97,000. Both are defensible; the point is that the task submits 97,000 tokens. Say which you are using.

12.7 Choosing, and composing

Reading-only; not scheduled in class.

The two answers are not rivals so much as answers at different scales, and the decision rule is short.

One GPU, or one node with a mixed workload: chunk. Disaggregation needs at least two instances to mean anything, and chunking costs 1.7% of memory traffic for essentially all of the benefit. There is no configuration in which a single-card deployment should not be chunking.

A cluster with fast interconnect and tight, differentiated SLOs: disaggregate. The independent parallelism, batch, and hardware choices are worth more than the transfer costs over NVLink, and the SLO decoupling is what lets you promise a TTFT and a TPOT separately rather than picking a point on one frontier.

Slow interconnect and large caches: chunk, and fix the cache first. The Ethernet/MHA cell in §12.5's table is where disaggregation stops paying; the right response is grouped-query attention or cache quantization (Oct 19), after which reconsider.

And they compose, which is the part the clean framing obscures. A disaggregated decode pool still receives new sequences and still needs some prefill work locally if it does any prefix recomputation; a prefill pool serving very long prompts still wants to chunk internally so that a 32K prompt does not monopolize an instance for two seconds. Production systems described in P/D-Serve do both, plus dynamic re-assignment of instances between roles as the traffic mix moves — which is the honest end state, and a good preview of Oct 14, where deciding which instance serves a request becomes the whole subject.

Discussion seeds

1. Who pays, exactly. §12.3 gave one request 32 ms of extra TTFT to give 164 requests their SLO back. Construct the workload where that trade is wrong — where the prefilling request is the one that matters most — and say what a scheduler would need to know to detect it.
2. The budget that moves. A fixed token budget makes later chunks of a long prompt slower, because S_ctx grows. Design a shrinking budget for a 32,768-token prompt that holds TPOT flat, and compute the resulting TTFT against a fixed budget of 512.
3. The transfer you would not make. At what KV size and link bandwidth would you refuse to disaggregate? Express it as a single inequality in bytes, bandwidth, and avoided interference, then check it against all six cells of §12.5's table.
4. Provisioning under uncertainty. You must buy a cluster today for traffic that is 80% chat and 20% agent, and the mix may invert within a year. Using §12.6's two ratios, what do you buy, and what would you rather build than buy?
5. Does prefix caching make disaggregation unnecessary? §12.6 showed a perfect prefix cache moves the agent ratio from 12:1 to 0.95:1. If Nov 4's cache works, does the case for separate pools weaken or strengthen? Argue both sides — the strongest version of each is not obvious.
6. The measurement you would demand. All three required-or-optional papers report improvements against a baseline they configured. Using Lecture 11 §11.6's table, name the baseline misconfiguration that would most inflate a disaggregation result, and the one number you would ask an author for to rule it out.

Key takeaways

Numbers worth memorizing

Quantity	Value	Source
Mixed-step spike at B = 164	20.1 ms → ~149 ms (7.4×)	§12.1
Prefill cost per token, S_ctx = 2,048	14.55 GFLOP (13.48 weights + 1.07 attention)	§12.3
Largest free chunk at a 20 ms SLO	T = 683 tokens	20.1 ms × 494.5 TFLOP/s ÷ 14.55 GFLOP
T = 512: TTFT paid / TPOT inflicted	161 ms (vs 129) / none	§12.3 table
Unchunked: TPOT penalty	+641%	149 ms vs 20.1 ms
Chunking overhead, T = 512 MHA / GQA-8	1.7% / 0.4%	S²/(2T) − S/2 re-reads
4K KV cache to ship	2.147 GB MHA, 0.537 GB GQA-8	Lecture 2 §2.15
Transfer: NVLink / PCIe / Ethernet (GQA-8)	1.3 / 8.4 / 21.5 ms	÷ 400, 64, 25 GB/s
Where disaggregation stops paying	MHA over 25 GB/s: 86 ms of 129 ms	§12.5
Pool ratio, chat vs agent	1.32 : 1 vs 11.97 : 1 (9×)	§12.6
Agent ratio with a perfect prefix cache	0.95 : 1	§12.6, previewing Nov 4

Self-check

Exercises

1. A shrinking token budget. For a 32,768-token prompt, a fixed T = 512 makes the last chunk far more expensive than the first, because its attention term scales with context. Compute the per-token cost at S_ctx = 1,024 and at 32,768, then design a budget schedule that holds step time at 20.1 ms throughout, and give the resulting TTFT against fixed-512. Solution sketch: Per-token cost = 13.48 + 0.524·(S_ctx/1,024) GFLOP. At S_ctx = 1,024: 14.0 GFLOP; at 32,768: 13.48 + 16.8 = 30.3 GFLOP, a 2.2× rise. Budget at fixed step time: T(S_ctx) = 9.94e12 ÷ cost, so 710 tokens early and 328 late. Integrate: total steps ≈ ∫dS/T(S) = ∫(13.48e9 + 0.512e6·S)dS ÷ 9.94e12 from 0 to 32,768 = (13.48e9·32,768 + 0.256e6·32,768²) ÷ 9.94e12 = (4.42e14 + 2.75e14)/9.94e12 = 72 steps → TTFT 1.45 s. Fixed 512: 64 chunks, but the later ones exceed the budget — the last costs 512·30.3e9/494.5e12 = 31.4 ms, so TPOT degrades 56% at the end of the prompt. The fixed budget does not hold the SLO it was chosen for, which is the practical reason engines expose the knob and long-context deployments have to tune it.
2. Price the whole interference bill. At Lecture 11 §11.6's operating point the engine retires and admits ~64 sequences per second. Assuming each admission brings a 4,096-token prompt and the engine does not chunk, compute the fraction of wall-clock spent in prefill-dominated steps and the effective TPOT a user experiences. Solution sketch: 64 prefills/s × 129 ms = 8.26 s of prefill per second — the engine is over-subscribed by 8.3×, so this workload is simply infeasible without more prefill capacity, which is §12.6's point arriving as a contradiction. Scale to a feasible 6 prefills/s: 6 × 129 ms = 0.774 s/s, so 77% of wall-clock is prefill-dominated. Effective TPOT = total step time ÷ tokens produced: in 1 s the engine spends 0.774 s on 6 mixed steps and 0.226 s on ~11 pure decode steps, producing 6·164 + 11·164 = 2,788 tokens, so TPOT ≈ 1/17 steps · … more simply, users see steps averaging (6·149 + 11·20.1)/17 = 65.6 ms, a 3.3× degradation. Chunking removes essentially all of it, since the prefill work is unchanged but distributed: 8 chunks × 6 prefills = 48 chunk-slots per second, comfortably inside ~50 steps/s.
3. The inequality for disaggregation. Write the condition under which shipping the cache beats sharing the card, in terms of KV bytes K, link bandwidth W, prefill time t_p, and the number of resident decode sequences B. Then evaluate it for all six cells of §12.5's table. Solution sketch: Sharing costs every resident sequence an extra t_p on one step, i.e. B·t_p of aggregate user-visible delay, and costs the prefilling request nothing extra. Disaggregating costs the prefilling request K/W of added TTFT and costs the residents nothing. So disaggregate when K/W < B·t_p if you weight all delay equally — at B = 164 and t_p = 129 ms the right-hand side is 21.2 s, and all six cells (max 85.9 ms) pass by two to four orders of magnitude. If instead you weight per-request latency equally, the condition is K/W < t_p, and five of six cells pass while MHA-over-Ethernet fails (85.9 vs 129 is a pass, marginally — recompute with a 2K prompt and t_p = 61 ms and it fails at 85.9). The choice of welfare function changes the answer by orders of magnitude, which is worth noticing before quoting any paper's improvement factor.
4. Provision for a mixed fleet. Traffic is 80% chat (P/G = 2,000/300) and 20% agent (97,000/1,600) by request count. Compute the blended prefill:decode ratio, then the ratio by task-weighted load, and say which you would provision against. Solution sketch: Per-request GPU-seconds: chat 0.0545 prefill / 0.0413 decode; agent 2.64 / 0.221. Blended over 100 requests: prefill = 80·0.0545 + 20·2.64 = 4.36 + 52.8 = 57.2 GPU-s; decode = 80·0.0413 + 20·0.221 = 3.30 + 4.42 = 7.72 GPU-s. Ratio 7.4 : 1. Note what happened: agents are 20% of requests and 92% of the prefill load, so the blended ratio sits near the agent ratio, not the chat one. Provision against 7.4:1, and understand that the number is almost entirely determined by the minority workload — so the variance in your traffic mix is the real risk, and the mitigation is prefix caching (which collapses the agent term) rather than more prefill GPUs.
5. Reconstruct a paper's baseline. A disaggregation paper reports 4.5× goodput over "vLLM". Using Lecture 11 §11.6's three regimes and §12.3's table, construct the most flattering baseline the authors could honestly have used, and compute how much of the 4.5× it explains. Then name the one configuration detail you would demand. Solution sketch: The most flattering honest baseline is a shared-pool engine with no chunked prefill and a batch large enough to make interference maximal. From §12.3, unchunked prefill inflates the step from 20.1 ms to 149 ms — 7.4× on TPOT — so if goodput is measured against a TPOT SLO, nearly all requests fail in the baseline and pass in the treatment, and a 4.5× is easily produced without the disaggregation doing any of the work. Enabling chunking in the baseline recovers essentially all of it (TPOT penalty → 0), leaving only the genuine gains from independent batch/parallelism/hardware, which are real but far smaller. The detail to demand is max_num_batched_tokens in the baseline — if it is unset or equal to the max sequence length, chunking was off and the comparison is against §12.1's problem rather than against the state of the art. This is Lecture 5 §5.3's "always measure a no-op" applied to somebody else's paper.

Reading guide

Required — Sarathi-Serve. Read §2–§3 for their framing of the throughput-latency tradeoff, which is Lecture 2 §2.16's frontier seen from inside an engine. Then the mechanisms, and keep the two separate as you read: chunked prefill is what you can do to a prompt, stall-free batching is what you must then do to every step, and the paper's contribution is really the second. The quantity to extract is the token budget — find where they discuss choosing it, and check their reasoning against §12.3's derivation of 683; you should be able to explain why the budget is a function of the decode batch and the SLO rather than a constant. In the evaluation, look for the shape of the TTFT-versus-TPOT curve as the budget varies rather than the headline numbers, and note which baselines have chunking disabled. Hold this question for the discussion: chunking hurts the prefilling request and helps every resident one — under what traffic mix does that stop being a good trade, and what would the scheduler need to know to notice?

Optional — DistServe. The opposing position, and the cleanest statement of why the two phases want different machines and different parallelism. Read the goodput-per-GPU metric definition carefully — it is a better metric than throughput for exactly Lecture 2 §2.16's reason — and the placement algorithm. Question: their gains come partly from removing interference and partly from per-phase optimization. Which half would chunked prefill also capture?

Optional — Splitwise. The hardware-heterogeneity argument: prefill wants FLOPs, decode wants bandwidth and capacity, so buy two SKUs. Read the phase characterization, which is the best empirical version of §12.1's table. Question: how much of their saving survives if you may only buy one SKU?

Optional — Mooncake. Disaggregation with the KV cache promoted to a cluster-wide storage tier — read the architecture figure now and save the rest for Oct 19, where it is required. Question: if the cache is addressable from both pools, does §12.5's transfer cost disappear or merely move?

Optional — MemServe and P/D-Serve. MemServe unifies the disaggregated and cached cases behind an elastic memory pool; P/D-Serve is what production does to all of this at scale, including dynamic re-assignment of instances between roles. Read P/D-Serve if you want the honest counterweight to three clean papers. Question: which of the four independent knobs in §12.4 survive contact with a real, shifting traffic mix?

Looking ahead

Monday (routing and load balancing, Oct 14) — after the holiday on Oct 25 and the Assignment 3 sharing session on Oct 14, with a guest speaker on Oct 19 — takes the question today's second answer forces. Once there is more than one instance, something must decide which one serves a request, and that decision now involves state: a replica holds a warm KV cache, so identical requests are not equally cheap everywhere. Llumnix goes further and migrates a running request between replicas, which is only possible because the cache is paged.

Assignment 3 (make the agent smarter, faster, cheaper) is due Oct 25, 11:59pm — the last assignment that lives entirely outside the serving stack — and Assignment 4 (serve your own agent) goes out that day, due Nov 10. A4 is where max_num_batched_tokens stops being documentation: §12.3's table is the experiment you will run, and its middle column is the answer you will report.

One thing to carry out of the room. Both answers today are versions of the same move — stop letting two workloads with opposite bottlenecks share a queue — and the whole disagreement is about the granularity at which you separate them: within a step, or across the cluster. That framing recurs on Oct 14 (which replica), Oct 19 (which memory tier), and Nov 18 (which program), and it is worth carrying as a question rather than an answer: what is sharing this resource that should not be?