INFERENCE · LAB 01 / 04
One request, two kinds of physics.
SPLIT INFERENCE · PREFILL ⊥ DECODE
Prefill is a momentary storm of compute; decode is a marathon of bandwidth.
Two utterly different physical processes live inside one inference request. Prefill processes your entire prompt in parallel in an instant — arithmetic intensity is high, a textbook compute-bound load. Decode generates token by token, and every step drags the whole weight matrix and KV cache through memory — arithmetic intensity collapses to a constant, pinned down by memory bandwidth.
Cramming them onto one chip means compute units idle through decode while bandwidth sits decorative through prefill — and worse, a new request's prefill storm interrupts everyone mid-decode. The direction pointed out by research like Splitwise is to split the two phases across hardware domains: a compute domain for the storms, a bandwidth domain for the marathons.
Drag the slider to change prompt length. Watch the amber stalls in the monolithic lane — that is a new request's prefill storm interrupting decode; in the split design decode never breaks stride. (Demo numbers; magnitudes from public research.)
Two arithmetic intensities
Prefill does hundreds of FLOPs per byte touched; decode rereads all weights for every token — two orders of magnitude apart.
Interference is latency
On a shared chip, one prefill storm stalls every running decode — the prime suspect behind tail-latency blowups.
Right hardware for each
High-FLOPs silicon for the compute domain, PIM and fat memory for the bandwidth domain — split, and both sides buy the right chips.