Users don’t experience your model’s benchmark scores. They experience the pause after they hit enter. LLM latency has a precise anatomy, and almost every “our AI feels slow” complaint traces to a handful of causes with known engineering fixes. Here they are, roughly in the order you should reach for them.
First, know which latency you have
LLM latency is two different numbers that get lumped together:
- Time to first token (TTFT) — how long before anything appears. This is dominated by prompt processing (the model must read your entire prompt before writing a word), plus network and queueing time.
- Tokens per second (TPS) — how fast text flows once it starts. This is dominated by the model’s generation speed on the serving hardware.
The distinction matters because the fixes are different — and because perceived speed lives almost entirely in TTFT. A response that starts instantly and streams steadily feels fast even when the full answer takes eight seconds; a response that arrives complete after four seconds of dead silence feels broken. Which leads to the cheapest fix on this list:
Fix 0: Stream, always
If your app waits for the complete response before showing anything, you’re voluntarily doubling or tripling perceived latency. Streaming costs nothing at the model layer — every serious API and serving engine supports it — and turns the interaction from wait → wall of text into instant acknowledgment → readable flow. If you do only one thing from this post, do this.
Fix 1: Shrink the prompt (the TTFT tax)
Prompt processing is roughly linear in the compute budget: a 20,000-token prompt takes meaningfully longer to start answering than a 2,000-token one. The usual offenders:
- Kitchen-sink RAG — stuffing ten retrieved chunks into the prompt when reranking would have found the right two. (This is yet another reason to measure retrieval properly.)
- Unbounded chat history — replaying the entire conversation every turn. Summarize or truncate old turns; the model rarely needs turn 3 verbatim by turn 40.
- Bloated system prompts — 3,000 tokens of accumulated instructions, of which the model demonstrably uses a fraction.
Halving the prompt roughly halves the time before the first word appears. Few optimizations anywhere in software pay this well for this little work.
Fix 2: Exploit the KV cache
While generating, a transformer stores attention state for every token it has seen — the KV cache — so each new token only needs to attend over stored state rather than reprocess the whole sequence. That’s an internal mechanic, but it has one big practical consequence: prefix reuse. If many requests share the same opening tokens (your system prompt, your few-shot examples, a long document being asked multiple questions), the serving layer can process that shared prefix once and reuse it — provider APIs sell this as prompt caching, often at a steep discount and with dramatically faster TTFT.
The engineering implication is almost embarrassingly simple: put the stable part of your prompt first and keep it byte-identical across requests. A timestamp or user name interpolated into the top of the system prompt silently destroys cache hits on every single call.
Fix 3: Cache whole answers, semantically
The fastest inference is the one you skip. Real applications see the same questions constantly — “what’s your refund policy?” arrives hundreds of times in slightly different words. A semantic cache embeds incoming queries and, when a new query lands close enough to a previously answered one, serves the stored answer in milliseconds instead of invoking the model. It’s the same embedding machinery RAG already uses, pointed at your own past traffic. For support-style workloads, hit rates of 20–40% are common — that fraction of your traffic becomes effectively free and instant.
Fix 4: Route by difficulty
Using a frontier model for everything is like dispatching a surgeon to apply band-aids. Small modern models are excellent at classification, extraction, reformatting and routine Q&A — at several times the speed and a fraction of the cost. The production pattern is routing: a fast, cheap model handles the request by default, and only genuinely hard cases escalate to the big model. Combined with self-hosted open models for the routine tier (here’s that stack, explained), this is the single biggest cost-and-latency lever in serious LLM systems — and it’s also how agent architectures stay affordable, since agents multiply model calls per user action.
Fix 5: Serving-layer throughput (when you self-host)
If you run your own models, two serving-engine features dominate throughput under load:
- Continuous batching — the GPU processes many requests in one batch and lets new arrivals join mid-flight rather than queueing behind the current batch. This is the headline trick of vLLM-class engines and routinely multiplies requests-per-GPU.
- Quantization — 4-bit weights don’t just shrink memory; they raise tokens-per-second, because generation speed is substantially memory-bandwidth-bound. Same GPU, noticeably faster model, minimal quality cost.
Under load, an unbatched server dies by queueing: each request may be fast, but the tenth user waits behind nine others. If your P50 looks fine and your P95 is a horror show, this is where to look.
The checklist
Stream the output. Cut the prompt. Keep stable prefixes byte-identical. Cache repeated answers semantically. Send easy work to small models. Batch and quantize if you serve your own. None of this is exotic — it’s the difference between an AI feature users trust and one they abandon, and it’s all standard engineering once you know the anatomy.
Have an LLM feature that works but drags? We do this for a living — the diagnosis usually takes a day, and fix number one is usually free.