INFERENCE · LAB 02 / 04
Memory, layered by temperature.
TIERED KV CACHE · HOT ⇅ COLD
The longer the context, the less reason every token has to live in the most expensive memory.
The KV cache is the session memory of autoregressive inference: every token leaves key-value pairs behind for future attention queries. By a few hundred thousand tokens of context, the KV cache outweighs the model itself — yet most tokens are never touched by attention most of the time.
The answer descends straight from OS page replacement: tier by access temperature. Fresh, frequently-hit KV blocks reside in the processing-in-memory arrays (joining attention at zero movement cost); long-unvisited blocks sink to CXL pooled memory; when an old topic resurfaces, its blocks are prefetched back to the hot tier.
The conversation keeps running: new KV blocks enter the context, age into the hot tier, then fall to the cold pool. Click "Reference something from 3 minutes ago" — and watch a forgotten block light up and ride back to the hot tier. (Demo numbers.)
Temperature is the tier
Hit frequency decides residence: hot blocks compute at zero distance, warm ones are milliseconds away, cold ones recall on demand — two orders of magnitude apart in cost.
Sinking is the norm
Attention is naturally sparse: in long contexts over 90% of KV blocks go unvisited for ages — keeping them hot is pure waste.
Prefetch by semantics
The Model-OS knows where the topic is heading — by the time an old thread resurfaces, its KV blocks are already on their way back.