Every number in Part II has been proportional to one quantity. The batching ceiling of 29, the 2.147 GB you migrate, the 0.537 GB you ship between pools, the 62.5 GB you allocate — all of them are 512 KiB per token, and today we attack it directly. There are four ways, and they are not variations on a theme: use fewer key/value heads, use fewer bits, keep fewer tokens, or read fewer of the tokens you kept. The first two shrink the cache, the third throws part of it away, and the fourth leaves it alone and touches less of it — which means they improve different metrics and compose differently. Today is also where Part II crosses a line it has respected for five meetings: paging, batching, chunking, migration and fusion were all exactly output-preserving, and half of today's techniques are not. By the end you should be able to place any cache optimization in the taxonomy, compute what it buys in concurrency or in TPOT, and say what evidence it owes you before you would deploy it.

Where this sits

Part II has spent six meetings making the cache cheaper to manage without making it smaller. Lecture 10 stopped over-reserving it, taking concurrency from 29 to 199. Oct 5 stopped wasting batch slots. Oct 7 stopped prefill and decode fighting over the device that holds it. Oct 14 learned to move it between replicas for 1.3 ms and to route requests toward it. Not one of those changed the fact that a token costs 512 KiB.

Today does, and it is the last meeting before the two that exploit the result: Nov 4 reuses cache across requests — which is a completely different axis from anything today, and the reason the two lectures do not overlap — and Nov 9 quantizes the weights, using machinery that today's §14.3 introduces for the cache. Today is also the pivot in the course's epistemics. Everything so far needed only a performance measurement to justify it. From today, half the techniques need a quality measurement on a frozen task set, which is Lecture 5 §5.3's instrument arriving in a serving lecture — and the reason Assignment 5, which goes out today, asks for both.

Instructor notes — Discussion plan

75-minute class, student-led. Two required papers pulling in opposite directions — one moves the cache, one shrinks it — so run it as two presentations with the taxonomy as the spine. The instructor owns §14.1's taxonomy, §14.7's composition arithmetic, and §14.8's line.

Time	Segment	Who	Notes
0–8	The 512 KiB bill and the four attacks	instructor	Protected. Board the taxonomy and the exact/approximate split before any paper.
8–14	§14.2 Fewer heads	instructor	Fast — they know GQA. The point is that you cannot apply it to a given checkpoint.
14–32	Presenter 1: KIVI	student	The outlier measurement is the paper. Force the "why do K and V differ?" question.
32–42	§14.4 Fewer tokens	room	H2O / SnapKV / sinks. Make the room name what evidence eviction owes.
42–50	§14.5 Fewer tokens read	instructor	The capacity-versus-bandwidth distinction; 1.6× TPOT and no concurrency gain.
50–64	Presenter 2: Mooncake	student	Read as storage. Tiers, transfer engine, SLO-aware rejection.
64–71	§14.7 What composes	instructor	Protected. The stack table and the honest total.
71–75	§14.8 The line, and A5	instructor	Exact vs approximate, and what A5 must report.

Questions to force. (1) "Which of the four attacks changes the model's output?" (2) "KIVI gives 3.2× capacity. Quest gives 1.6× TPOT. Which do you want, and what does the question depend on?" (3) "You have a checkpoint someone else trained. Which attacks are available to you?" (4) "What would you have to measure before shipping eviction to a paying customer?"

Reading-only, not scheduled: §14.6's CacheGen and ring-attention material, and §14.7's second table.

If running long: compress §14.5 to its one number and drop §14.6 to a sentence. Never cut §14.1 or §14.7 — the taxonomy is what makes the reading list navigable, and the composition table is what stops students multiplying factors that do not multiply.

Learning objectives

By the end of this class you should be able to:

1. Write the KV-cache formula and name every term you could attack, with the technique family that attacks it.
2. Distinguish the four attacks by which metric they improve — concurrency, TPOT, or capacity elsewhere — and say why that matters more than their headline factors.
3. Explain why keys and values want different quantization axes, and compute the effective bits of a grouped quantization scheme including its scales.
4. Compute the concurrency and TPOT consequences of a given cache reduction, and say which of the two the deployment actually needed.
5. Explain what an attention sink is and why dropping the first few tokens is worse than dropping middle ones.
6. Distinguish reducing cache size from reducing cache traffic, and compute each one's effect on a decode step.
7. Compose several cache optimizations and say which factors multiply, which overlap, and which conflict.
8. State what evidence an exact optimization owes you and what an approximate one owes you, and design the smaller of the two experiments.

14.1 One formula, four places to attack it

Everything today follows from reading Lecture 2 §2.15's formula as a list of targets rather than as a fact.

The 2 is K and V and is not negotiable. L and d_head are the model's shape. That leaves four quantities a serving system can move, and each has a literature.

n_kv — fewer key/value heads. Multi-query and grouped-query attention. A pretraining decision, so it is available when you choose a model and not when you are handed one. §14.2.

b — fewer bits per value. Quantize the cache itself: KIVI and its relatives. §14.3.

S — fewer tokens retained. Eviction and compression: H2O, SnapKV, StreamingLLM. This is the one that discards information. §14.4.

Nothing in the formula — fewer tokens read. Keep the whole cache and touch only the part that matters this step: Quest, InfiniGen. It changes no term above, which is exactly why it improves a different metric. §14.5.

And a fifth move that is not a reduction at all: put the bytes somewhere else. Mooncake's tiered store, CacheGen's compressed bitstream. §14.6.

Two organizing distinctions matter more than any individual factor, and the lecture is arranged around them.

Capacity or bandwidth? A decode step's cost is (13.5 GB of weights + B × per-sequence KV) ÷ bandwidth. Shrinking KV bytes per token helps both terms of what you care about: more sequences fit (concurrency) and each step moves less (TPOT). Reading fewer tokens helps only the second. Moving the cache off the GPU helps only the first, and hurts the second. These are not interchangeable wins, and which one you want depends on whether your deployment is short of memory or short of time.

Exact or approximate? Every optimization in Part II so far — paging, continuous batching, chunked prefill, migration, kernel fusion — produced bit-comparable outputs, so justifying it required only a performance measurement. Fewer heads is exact for the model you trained. Everything else today is approximate: quantization perturbs the values, eviction removes them, sparse reading ignores them. Approximate techniques need a quality measurement, and Lecture 5 §5.3 already told you what a credible one looks like — a frozen task set, a frozen criterion, a frozen scorer, and an interval on the result. §14.8 is that argument.

Instructor notes

Minutes: 8. Protected. Instructor holds this; no paper yet. Board: The formula, then four arrows out of it labelled n_kv / b / S / (nothing), with the family names. Then two lines underneath: "capacity vs bandwidth" and "exact vs approximate". Ask the room: "Which of the four is available for a checkpoint you did not train?" Everything except n_kv. This reframes the reading list as a menu. Expect confusion: Students expect the four factors to multiply into one big number. §14.7 is the antidote; promise it now.

14.2 Fewer heads, and the decision you already missed

Grouped-query attention shares one key/value head among several query heads, so n_kv falls while the query heads and nearly all the FLOPs stay. Lecture 2 §2.15 priced it: n_kv = 32 → 8 takes the cache from 512 KiB to 128 KiB per token, a clean 4×, which takes concurrency at 4K from 29 to 116 and quadruples throughput at identical TPOT. Multi-query attention is the limit n_kv = 1, worth 32× on paper and generally too aggressive for quality.

Three things about this are worth saying, none of which is the factor of four.

It is exact, and that is unusual here. GQA does not approximate a model with 32 KV heads; it is a different architecture, trained as such. Once trained, its outputs are its own and nothing is being degraded at serving time. That puts it on the far side of §14.8's line from everything else today, and it is why it ships universally with no flags and no quality caveats.

It is not available to you. This is the practical point. n_kv is fixed at pretraining, so if you are handed a checkpoint with 32 KV heads, no serving technique gives you GQA's 4×. There is research on post-hoc head merging, and it is exactly as approximate as everything else in this lecture. The cheapest 4× in LLM serving is a decision made months before the serving team is consulted, which is worth knowing when you choose a model for Assignment 4.

Its effect keeps reappearing, which is the real lesson. Count the places this semester where GQA moved a threshold rather than a number: the batching ceiling (29 → 116, Lecture 2), the swap-versus-recompute break-even (33.3 → 8.3 GB/s, Lecture 10 §10.5), the disaggregation handover (85.9 → 21.5 ms over Ethernet, Lecture 12 §12.5), the migration cost (5.4 → 1.3 ms, Lecture 17 §17.4), and the fetch-versus-recompute bar (16.6 → 4.2 GB/s, Lecture 13). One architectural choice moved five independent serving decisions, because they were all proportional to the same quantity. That is what it means for a number to be load-bearing, and it is the best argument in the course for reading a model's config file before designing anything around it.

Instructor notes

Minutes: 6. Keep it fast — they know the mechanism from Lecture 2. Board: "512 → 128 KiB, 29 → 116" then the list of five thresholds GQA moved. Ask the room: "You have a 32-KV-head checkpoint and need 4×. What are your options?" All approximate. That lands §14.1's second distinction.

14.3 Fewer bits, and why K and V are not the same tensor

The cache is a tensor of activations, so quantize it. The naive version — round everything to int4 — degrades badly, and KIVI's contribution is the measurement explaining why, plus the design that follows from it.

The observation: keys have persistent per-channel outliers and values do not. In the key tensor, certain channels of d_head carry consistently large magnitudes across all positions; in the value tensor, magnitudes are distributed without that structure. Quantization error is dominated by the range of whatever group shares a scale factor, so the two tensors want their groups drawn along different axes: keys per-channel (each channel gets its own scale, so one wild channel cannot inflate the range of the rest) and values per-token (grouping along the head dimension within a token). One tensor, two treatments, derived from where the outliers actually are — which is why the paper is worth reading as an empirical exercise rather than an algorithmic one.

Now price it honestly, which means counting the scales.

Two caveats, both of which the headline factor hides.

The scales are not free and neither is the dequantization. The attention kernel must now dequantize the cache as it reads it, and Lecture 15 will do this accounting properly for weights: dequant work lands inside the kernel's inner loop, and at low arithmetic intensity that is where you can least afford it. Decode attention is 1 FLOP/byte (Lecture 9 §9.8), so it has no spare arithmetic — the saving is real because bytes are the binding constraint, but it is smaller than 3.2× and the honest figure comes from a measurement of your kernel, not from a byte count. This is the same trap Lecture 15 identifies for weight quantization, where a 4-bit checkpoint can serve slower than an 8-bit one.

It is approximate, so it owes you a quality measurement. KIVI's tables show which tasks degrade, and the pattern is the important part: long-context retrieval tasks are more sensitive than short generative ones, because a quantization error in a key changes which token gets attended to, not merely how much. That is a qualitatively worse failure than a slightly noisier output, and it connects directly to Lecture 4 §4.3's warning that a context miss is silent — a corrupted key does not raise an error, it produces a fluent answer about the wrong retrieved fact.

Instructor notes

Minutes: 18, presenter-led. Board (presenter): The key tensor drawn as a grid with two dark columns; the value tensor with scattered dark cells. That picture is the paper. Force the question: "Why can't one scheme serve both tensors?" Push until they say the outliers have different geometry. Instructor adds: the 4 + 32/g arithmetic. Students will quote 4× for int4; make them derive 3.2×. Expect confusion: "Quantization is free because bytes are the bottleneck." Bytes are the bottleneck; dequant still lands in the kernel. Promise Nov 9.

14.4 Fewer tokens: eviction, and the tokens you must not evict

The third attack discards cache entries. It is the most effective per byte and the most dangerous, and the three well-known variants differ in when they decide and what they use to decide.

H2O observes that attention mass concentrates on a small subset of positions — "heavy hitters" — and that the set is fairly stable over decoding. Keep the heavy hitters plus a recent window; evict the rest. The decision is made continuously, from accumulated attention scores.

SnapKV notices that the prompt's own attention pattern, observed at the end of prefill, predicts which prompt tokens later generation will need. So compress the prompt's cache once, at the prefill boundary, which is cheap and well-timed: the biggest cache in an agent workload is the prompt, and prefill is when you have just computed everything you need to score it.

StreamingLLM contributes the most surprising fact, and it is worth understanding rather than memorizing. If you keep a sliding window of recent tokens and drop everything older, quality collapses — and it is rescued by retaining just the first few tokens of the sequence, which StreamingLLM calls attention sinks. The mechanism is a property of softmax: it must distribute a total mass of one, so when no position is genuinely relevant the model needs somewhere to put the leftover attention, and it learns to dump it on the earliest positions. Remove them and that mass is forced onto tokens that are semantically relevant, distorting the distribution everywhere. Four tokens, kept for a structural reason rather than a semantic one, are the difference between a working sliding window and a broken one.

That last result is the best illustration of why this family is difficult. The naive intuition — "drop the least-attended tokens" — is exactly wrong for the first four positions, which are attended a lot and carry no information. Attention weight is a poor proxy for importance in both directions.

So this family's honest position is: enormous factors, available on any checkpoint, and a failure mode that no performance benchmark can detect. It is the clearest case in the course for the frozen task set of Lecture 5 §5.3, and §14.8 says what that means in practice.

Instructor notes

Minutes: 10, room-led. Board: A sequence as a bar; shade a recent window; then add four cells at the far left labelled SINK. Ask why they are there before explaining. Ask the room: "Attention sinks get a lot of attention weight and carry no information. What does that tell you about using attention weight as an importance score?" That it is a proxy, and a bad one at the boundaries. Expect confusion: Students think sinks are semantic (a BOS token's meaning). They are structural — softmax needs a sink. Say it twice.

14.5 Fewer tokens read, and putting the bytes elsewhere — Wednesday's subject

Previewed here because §14.7's composition arithmetic needs them; Wednesday, Oct 14 is the full treatment, with Mooncake and Quest as its required readings.

Two families remain, and both differ from §14.2–§14.4 in a way §14.1 already flagged: they do not shrink the cache.

Read fewer of the tokens you kept. Quest and InfiniGen keep the entire cache resident and attend over only the pages a cheap importance estimate says matter this step. This changes no term in §14.1's formula, so it buys no concurrency at all — and it reduces the bytes a step moves, so it buys TPOT. The diagnostic is simple and worth memorizing now: ask how many sequences fit. If the answer is unchanged, you are looking at a bandwidth technique, not a capacity one.

Move the bytes somewhere else. Mooncake promotes the cache from a per-GPU array to a tiered, cluster-wide store — HBM, host DRAM, remote memory — at which point the design question stops being "how big is the cache?" and becomes "which tier is each block in, and what does a miss cost?" CacheGen compresses the cache into a streamable bitstream for when the tier you want is too slow. Both buy capacity elsewhere and cost TPOT, which is the opposite sign from everything above.

The break-even that governs the second family is one this course has now derived four times, and it is the same inequality each time — fetch beats recompute when KV bytes ÷ link bandwidth is less than prefill FLOPs ÷ achieved FLOP/s, i.e. above 16.6 GB/s at MHA and 4.2 GB/s at GQA-8 for a 4,096-token prefix. Lecture 10 §10.5 used it for swapping to host memory, Lecture 12 §12.5 for the disaggregation handover, Lecture 17 §17.4 for migration, and Lecture 13 will use it for a remote prefix store. Four mechanisms, one inequality, and the reason the answer is usually "move it" is that KV bytes are linear in S while prefill FLOPs are quadratic.

Instructor notes

Minutes: 5 — a preview, not a segment. The full versions are Wednesday. Board: "read less → TPOT only" over "move it → capacity elsewhere, TPOT worse". Then the four-mechanisms-one-inequality line. Ask the room: "How do you tell a capacity technique from a bandwidth one in one question?" How many sequences fit. Expect confusion: Sparse reading gets filed with eviction because both "skip tokens". One deletes and one declines to look; only the first frees memory.

14.7 What composes, and what does not

The four attacks have headline factors of 4×, 3.2×, 2–4×, and 1.6–2.6×, and it is tempting to multiply them. Do not. Build the table instead.

	Attacks	Improves	Composes with
GQA-8	n_kv	concurrency and TPOT	everything (it is a different model)
KV int4, g = 32	b	concurrency and TPOT	GQA; overlaps eviction
Eviction, ρ = 0.5	S	concurrency and TPOT	GQA, quantization
Sparse read, 25%	bytes touched	TPOT only	all — but its factor is bounded by the others
Tiering	where bytes live	capacity elsewhere	all; hurts TPOT

Three rules fall out, and they are what a student should leave with.

Size reductions multiply, because they are independent factors on the same product. GQA-8 × int4(g=32) × ρ=0.5 is 4 × 3.2 × 2 = 25.6× on bytes per token: 512 KiB → 20 KiB, and concurrency at 4K from 29 to 744. That multiplication is legitimate — they touch different terms of the formula.

Sparse reading does not multiply with them; it competes with them. Its gain is a fraction of the cache term of the step, and every size reduction shrinks that term, so the two eat the same lunch. Once the cache is 20 KiB/token, a step at B = 744 moves 13.5 + 744 × 0.0126 = 22.9 GB, of which weights are 59% — so reading 25% of the cache now buys just 1.4× rather than 2.6×. The more successful your size reductions, the less sparse reading has left to give, and the weights become the next problem, which is precisely Lecture 15's subject.

Approximations do not compose in quality the way they compose in bytes. Quantization perturbs values, eviction removes them, and sparse reading ignores them, and stacking all three means a token may be quantized, and then evicted, and then not read. Their error budgets interact, and no paper evaluates the stack you are about to build. So the composite factor is arithmetic you can trust and the composite quality is not — which is the entire content of the next section.

14.8 Exact and approximate need different evidence

Part II crossed a line today, and the discipline it demands is worth stating explicitly because it is the thing most easily skipped.

An exact optimization — paging, continuous batching, chunked prefill, migration, fusion, and GQA as an architecture — cannot change an output. Justifying it requires a performance measurement: a before, an after, a stated configuration, and percentiles. If it is faster it is better, and there is nothing else to check.

An approximate optimization — cache quantization, eviction, sparse reading, compressed transport — can change every output. Justifying it requires the performance measurement and a quality measurement, and Lecture 5 §5.3 already specified what a credible one is: a frozen task set, a frozen criterion, and a frozen scorer, all fixed before you tune anything, with an interval on the result. Lecture 5's arithmetic applies unchanged — 20 tasks gives you ±20 points, and halving that interval costs 4× the tasks — so a two-point quality claim on a 50-task suite is not evidence.

Two task-selection points specific to cache techniques, because a generic benchmark will systematically miss their failures. Long-context retrieval is the sensitive case: an error in a key changes which token is attended to, so needle-in-a-haystack and multi-document QA degrade long before perplexity or short-form generation does. And the failure is silent — Lecture 4 §4.3's point, arriving one layer down. A cache optimization does not raise an error or produce gibberish; it produces a confident answer about the wrong retrieved fact, which is the single hardest failure mode to detect from the outside and the reason it must be measured deliberately rather than noticed in production.

This is also why Assignment 5, which goes out today, asks for both numbers. If you enable any technique from §14.3–§14.6 in A5, the write-up needs the speedup and the quality result on your frozen set, with the interval — and a technique you cannot show is quality-neutral is a technique you have not finished evaluating.

Discussion seeds

1. The checkpoint you were handed. You must serve a 32-KV-head model and need 4× more concurrency. Rank §14.3–§14.5 by what you would try first, and justify the order by evidence cost rather than by factor size.
2. Attention sinks as a warning. StreamingLLM keeps four tokens for a structural reason. What other "obviously unimportant" state in this course turns out to be load-bearing, and what does that pattern suggest about designing eviction policies from first principles?
3. The composite nobody evaluated. §14.7's stack is 25.6× and no paper evaluates it. Design the smallest experiment that would make you comfortable shipping it, and estimate its cost in GPU-hours using Lecture 5's sample-size arithmetic.
4. Capacity or time? Construct two deployments — one where KIVI's 3.2× is worth more than Quest's 2.6×, and one where the reverse holds. What single property of the workload decides?
5. Tiering against reuse. Mooncake moves the cache out of the GPU; Nov 4 reuses it across requests. Do these compete for the same memory, and if so, what is the allocation rule?
6. What the survey would say about your A4. Take your Assignment 4 deployment and place it in §14.1's taxonomy: which of the four terms have you already attacked, knowingly or by default? What is your cheapest remaining factor, and what evidence does it owe?

Key takeaways

Numbers worth memorizing

Quantity	Value	Source
KV per token	512 KiB MHA, 128 KiB GQA-8	2·L·n_kv·d_head·b
Concurrency at 4K	29 MHA, 116 GQA-8	62.5 GB ÷ 2.147 (0.537) GB
Effective bits, int4 at g = 32	4 + 32/32 = 5 → 3.2×, not 4×	§14.3
KIVI on GQA-8	128 → 40 KiB/token; B = 116 → 372	§14.3
Eviction ρ = 0.5 / 0.25	B = 116 → 232 / 466	§14.4
Attention sinks needed	4 tokens, kept structurally not semantically	StreamingLLM
Sparse read 25%, 640-tok sequences	26.9 → 16.85 GB/step: 1.6× TPOT, 0× concurrency	§14.5
Sparse read 25%, 16K sequences	75.8 → 29.1 GB/step: 2.6× TPOT	§14.5
Fetch-vs-recompute break-even	16.6 GB/s MHA, 4.2 GB/s GQA-8	2.147 (0.537) GB ÷ 129 ms
Full stack: GQA × int4 × ρ=0.5	25.6×; 512 → 20 KiB; B = 29 → 744	§14.7
Step composition after the stack	59% weights — hand off to Nov 9	§14.7

Self-check

Exercises

1. Reach a target concurrency on a checkpoint you cannot change. You are handed a 32-KV-head model and must serve 300 concurrent 4,096-token sequences on one H100. Enumerate the combinations of §14.3–§14.4 that reach it, and rank them by the evidence each would require. Solution sketch: Need 62.5 GB ÷ 300 = 0.208 GB per sequence = 52 KiB/token, down from 512 KiB — a 9.85× reduction, and GQA is unavailable. Options: int2 at g = 32 (3 bits, 5.33×) plus ρ = 0.55 eviction → 9.7×, marginal; int4 at g = 128 (4.25 bits, 3.76×) plus ρ = 0.38 → 9.9×; int4 at g = 32 (3.2×) plus ρ = 0.32 → 10.0×. All three need aggressive eviction, so the ranking is by evidence cost: quantization has published per-task tables you can check against your own workload, whereas eviction at ρ = 0.32 is well beyond most papers' evaluated range and its failure is silent. The honest answer is that 300 is not reachable on this checkpoint with evidence you can obtain cheaply — and that the right move is to change the model, which is §14.2's real point and Exercise 1's intended punchline.
2. Where sparse reading stops paying. For GQA-8, compute the TPOT gain from reading 25% of the cache at context lengths 512, 4,096, and 32,768 at the largest B memory permits in each case. Then state the general rule. Solution sketch: Per-token 128 KiB. 512 tokens: 0.067 GB/seq, B = 932; step = 13.5 + 62.5 = 76.0 GB; at 25% of cache: 13.5 + 15.6 = 29.1 GB → 2.6×. 4,096: 0.537 GB, B = 116; step 13.5 + 62.3 = 75.8; sparse 13.5 + 15.6 = 29.1 → 2.6×. 32,768: 4.295 GB, B = 14; step 13.5 + 60.1 = 73.6; sparse 13.5 + 15.0 = 28.5 → 2.6×. The gain is the same, because at the memory-limited B the cache always fills the 62.5 GB budget regardless of context length — so the ratio (13.5 + 62.5)/(13.5 + 15.6) is context-independent. The §14.5 result differs because it held B = 164 fixed rather than filling memory. The general rule: sparse reading's gain depends on the cache's share of the step, which is set by how full memory is, not by context length per se — which is a cleaner statement than the literature usually gives, and it means the technique pays whenever you are running near the memory ceiling.
3. Price the dequantization you were told to ignore. Suppose dequantizing int4 KV costs 4 instructions per value on a unit that retires 33.5e12 instructions/s (Lecture 15's illustrative board model). For GQA-8 int4 at B = 372, 4,096-token sequences, compute the dequant time per step and compare it with the step's memory time. Then say at what B dequant becomes the bottleneck. Solution sketch: Values per step = B · S · 2 · L · n_kv · d_head = 372 · 4,096 · 2 · 32 · 8 · 128 ÷ … more simply, bytes ÷ 0.5 B per int4 value: cache bytes = 372 × 0.168 GB = 62.5 GB → 125e9 values. Dequant = 4 × 125e9 ÷ 33.5e12 = 14.9 ms. Memory time = (13.5 + 62.5) ÷ 3,350 = 22.7 ms. So dequant is 66% of the memory time and mostly hidden if the kernel overlaps them — but it is the same order of magnitude, which is the warning. Dequant becomes binding when 4 · (bytes ÷ 0.5) ÷ 33.5e12 > (13.5e9 + bytes) ÷ 3.35e12, i.e. when 8·bytes/33.5 > (13.5 + bytes)/3.35 → 0.239·bytes > 4.03 + 0.299·bytes — which has no positive solution, so with these constants dequant never dominates outright. But the margin is thin enough (66%) that a kernel with poor overlap will lose most of the 3.2×, which is exactly Lecture 15's thesis arriving early, and the reason §14.3 insists the honest figure is a measurement.
4. Design the experiment §14.7 needs. You intend to ship GQA-8 × int4(g=32) × ρ=0.5. Using Lecture 5 §5.3's arithmetic, size the quality experiment needed to detect a 3-point drop in accuracy with 95% confidence, name the task mix, and estimate the GPU cost. Solution sketch: For a proportion near 0.7, the 95% interval half-width is 1.96·√(0.7·0.3/n) ≈ 0.9/√n; to resolve 3 points we need half-width ≤ 1.5 points → √n ≥ 60 → n ≈ 3,600 tasks for an unpaired comparison. Paired (same tasks, both configurations) is far cheaper: with a per-task paired difference and Lecture 5's 2σ/√n ≤ 3 rule at σ ≈ 15 points, n ≈ 100 tasks, run under both configurations. Mix: at least half long-context retrieval (needle-in-a-haystack at 4K/16K/32K, multi-document QA) since §14.8 says that is where the sensitivity is, plus short-form generation and one code task as controls. Cost: 200 runs × ~2.64 GPU-s of prefill for a long-context task ≈ 9 GPU-minutes of compute, so the experiment is nearly free and the reason it is skipped is that nobody freezes the set, not that it is expensive. That is the exercise's point.
5. Which term binds after you win. Apply §14.7's full 25.6× stack and then find, for a 20 ms TPOT SLO, the largest B and the resulting throughput. Compare with the pre-stack numbers and say what the next optimization must attack. Solution sketch: Post-stack KV = 20 KiB/token; at 4,096 tokens that is 0.082 GB/seq. SLO: 20 ms × 3,350 GB/s = 67.0 GB per step; minus 13.5 GB of weights leaves 53.5 GB → B ≤ 652. Memory would allow 62.5 ÷ 0.082 = 762, so the SLO binds again (as it has since Lecture 10). Throughput = 652 tokens per 20 ms = 32,600 tok/s, against Lecture 11's 8,159 — a 4.0×. Step composition: 13.5 of 67.0 GB = 20% weights at this B… but note the per-sequence term is now tiny, so the marginal cost of another sequence is 0.082 GB while the fixed cost is 13.5 GB. The next optimization must attack the weights — either their bytes (quantization, Nov 9) or how many tokens each read produces (speculative decoding, Nov 11). That is the correct hand-off and it is why those two lectures come next.

Reading guide

Required — Mooncake. Read it as a storage paper and the design will make sense immediately. Start with the architecture figure — you met it on Oct 7 — then the KV-cache-store sections: what a block is, which tiers exist, and what the transfer engine has to guarantee. The material worth the most attention is the policy: how blocks are placed and evicted across tiers, and the SLO-aware early rejection, which is the first admission policy in this course that refuses work to protect latency rather than memory. Skim the cluster-operations detail. Hold this question: §14.6's break-even says fetching beats recomputing above 4.2 GB/s at GQA-8 — go through their tiers and decide which ones clear that bar, and what they should do with the ones that do not.

Required — KIVI. Read the analysis section first and slowly: the per-channel-versus-per-token asymmetry is the entire paper, and everything else is engineering downstream of it. Make sure you can state why the two tensors differ rather than merely that they do. Then the algorithm, checking the effective-bit arithmetic against §14.3's 5-bits-at-g-32 — the paper's factors and yours should agree, and if they do not you have missed a scale somewhere. Then the accuracy tables, read for which tasks move: expect long-context and retrieval-flavoured tasks to be the sensitive ones, and ask whether the evaluated set would have caught a problem on your own workload. Hold this question: this is Part II's first output-changing optimization — write down the experiment you would demand before enabling it, and check how much of that experiment the paper actually ran.

Optional — SnapKV. Compress the prompt's cache at the prefill boundary, using the prompt's own attention pattern. Question: why is prefill-time the right moment, and what does that imply for an agent workload whose prompt grows every step?

Optional — Quest and InfiniGen. The fewer-tokens-read family. Read for the importance-estimation mechanism and its cost on the critical path. Question: §14.5 argued a wrong estimate here is recoverable while a wrong eviction is not — do the papers make that argument, and should it change how aggressive you are willing to be?

Optional — CacheGen. Compression for transport rather than for residency. Question: how does paying the distortion once per fetch rather than once per read change the acceptable error budget?

Optional — Ring attention. The other answer to long context: distribute the sequence, not the cache. Question: which of today's four attacks does it make unnecessary, and which does it make more important?

Optional — HiSparse, prediction-based management, and the survey. Start with the survey if you are presenting — it is the fastest way to see whether a technique you are proposing is new. Question: does the survey's taxonomy match §14.1's four terms, and if not, what is it organizing by instead?

Looking ahead

Wednesday (Oct 28) is the sharing session for Assignment 4 — bring your stack, the quality gap you closed against the frontier API, and what it cost you. Assignment 5 goes out today, due Dec 2, and §14.8 is its evidentiary standard: if you enable anything from today, report the speedup and the quality result on a frozen set, with the interval.

Then Monday, Nov 4 (prefix cache) attacks the same bill along an axis nothing today touched. Everything in this lecture made a single sequence's cache cheaper; Nov 4 observes that different requests' caches are often identical — Lecture 5's ledger says 92% of an agent's prompt tokens are re-sent — and that identical prefixes have identical caches by a property of causal attention. That is a 12.6× on prefill, orthogonal to today's 25.6× on bytes, and it is where the the final project is already under way. Monday, Nov 9 (pruning and quantization) then takes the term §14.7 left binding: after the stack, 59% of a step is weights.

One thing to carry out of the room. Today was the first time this course asked you to accept a worse model in exchange for a faster one, and the arithmetic for the trade was easy while the evidence for it was not. Every technique from here on has that shape. The habit worth forming is the one Lecture 5 insisted on and §14.8 restates: freeze the set before you tune, measure the thing that could break rather than the thing that is easy to measure, and treat a silent failure as the default outcome rather than the unlucky one.