Module 5 of 6 · Expert · 9 min read
Inference optimisation
KV caches, batching, quantisation, speculative decoding and prompt caching.
- 1Decoding is memory-bound; batching uses the GPU better.
- 2Quantisation shrinks models with modest quality loss.
- 3Prompt caching is the biggest cost lever for API users.
01Why inference is expensive
Generating each token requires reading the model's weights from memory. For large models this is limited by memory bandwidth, not raw compute. Serving systems batch many requests together so each weight read is shared.
The KV cache stores attention keys and values for previous tokens so they are not recomputed; it grows with context length and is a major memory cost.
02Optimisation techniques
Serving frameworks such as vLLM, SGLang and TensorRT-LLM combine many of these.
- Quantisation: store weights in 8, 4 or fewer bits.
- Speculative decoding: a small draft model proposes tokens that the large model verifies in parallel.
- Continuous batching and paged attention for high throughput.
- Prefix caching: reuse computation for shared prompt prefixes.
03For API users
Structure prompts so stable content (system prompt, tools, documents) comes first and varies last, then enable prompt caching. Use batch endpoints for offline work, and route simple traffic to smaller models.
Finished reading?
Mark it complete to track your progress, then test yourself with the LLM Internals quiz.