Pallavi Ahuja

Pallavi AhujaSep 25, 2026

KV cache looks like a small implementation detail in a Transformer diagram. In a serving system, it can become one of the main consumers of GPU memory. The reason is simple: model weights are mostly fixed during inference, but the state associated with an active request keeps growing. Every cached token leaves behind key and value tensors that later tokens may need. Those tensors exist across the attention layers, and they remain alive for as long as that part of the context is being used. So context length is not just a number in the model specification. It has a memory cost. Concurrency has one too. If many requests are decoding at once, each carries its own KV state. Longer sequences mean larger caches, and more simultaneous sequences mean more caches competing for the same GPU memory. Architecture choices show up here very quickly. GQA and MQA reduce the number of KV heads. KV quantization reduces the bytes used by the stored states. Sliding-window attention can limit how much history needs to remain available when the model is designed for it. And caching does not make attention over the existing context disappear. It saves the work of recomputing old K/V projections, but with full attention, each new query still has to use the relevant cached keys and values. This is why KV cache sits right in the middle of context length, memory capacity, decode cost and serving concurrency. It is a small piece of the attention equation with very large systems consequences.

Pallavi AhujaSep 26, 2026

Mathematics would feel very different if we learned more of it like this. Frequency and phase are easy to define on paper, but much easier to understand once you can see what changing them actually does. These are Lissajous curves. They are made by taking two sine waves and using one as the x-coordinate and the other as the y-coordinate. Change the ratio between their frequencies, and you get a different pattern. With rational frequency ratios such as 1:2, 2:3 or 3:4, the trajectory repeats and closes on itself. Keep the frequencies equal and change only the phase difference, and you can watch a line open into an ellipse, become a circle at 90° for equal amplitudes, and eventually collapse into the opposite diagonal. The equations did not suddenly become simpler. You just gained a picture for what they mean. A lot of mathematics becomes easier once the symbols are connected to something you can see.

Pallavi AhujaSep 28, 2026

I put together a handbook on how quantization actually works, starting with affine integer quantization => how floating-point values are mapped onto a smaller set of representable codes using scales, zero-points, rounding, clipping, and different quantization granularities. It works through symmetric and asymmetric quantization, per-tensor/per-channel/per-group scaling, calibration, PTQ and QAT, weight-only and activation quantization, outliers, GPTQ, AWQ, SmoothQuant, LLM.int8(), NF4 and QLoRA, FP8, KV-cache quantization, KIVI, KVQuant, and the kernel/hardware side of making low precision useful. Made it to be the explanation I wish I'd had when I first tried to understand what people actually mean when they call a model "4-bit," and why fewer bits can reduce memory without automatically making inference faster.