Open-weight models have gotten genuinely good — good enough that “run it on our own hardware” is now a serious engineering option, not a hobbyist stunt. But the tooling names fly around interchangeably, and they shouldn’t: llama.cpp, Ollama and vLLM solve three different problems. Here’s the map.
The three tools, and what each actually is
llama.cpp — the inference engine. A C/C++ implementation of LLM inference built for efficiency on ordinary hardware: CPUs, Apple Silicon, consumer GPUs. Its superpower is running models smaller than they were trained through quantization (more below), using the GGUF file format that has become the de facto standard for local models. It’s fast, dependency-free and runs on almost anything — but it’s a low-level tool. You get a library and a command line, not an experience.
Ollama — the local runner experience. Ollama wraps an inference engine (llama.cpp lineage) in the thing developers actually want: ollama run llama3 and you’re chatting. It handles model downloads, versions, GPU offload configuration, and exposes a local HTTP API that mimics the OpenAI format — so any app written against an LLM API can point at your own machine instead. For development, prototyping and single-machine deployments, this is the tool.
vLLM — the production server. vLLM answers a different question: how do you serve one model to hundreds of concurrent users on a real GPU server? Its core innovations — PagedAttention (treating GPU attention memory like virtual memory) and continuous batching (new requests join the batch mid-flight instead of waiting for the previous batch to finish) — routinely deliver several times the throughput of naive serving. When self-hosting graduates from “runs on my machine” to “serves the company,” this is the class of tool you move to.
One sentence each, for the whiteboard: llama.cpp runs models efficiently, Ollama makes running them pleasant, vLLM makes serving them scale.
Quantization, in plain terms
A model’s weights are stored as numbers, and precision costs memory. Full precision spends 16 bits per weight; quantization stores approximations in 8, 5 or 4 bits. A 70-billion-parameter model needs ~140 GB at full precision — and ~40 GB at 4-bit. That’s the entire trick that makes local LLMs possible.
The counterintuitive part is how little quality it costs. A well-quantized 4-bit model typically loses only a few percent on benchmarks — usually imperceptible in normal use — because the relationships between weights survive even when individual precision drops. Below 4-bit, degradation gets noticeable; 4-bit (the Q4_K_M you’ll see on GGUF files) is the sweet spot most local deployments choose.
Rough VRAM arithmetic for planning: a 4-bit model needs about 0.6 GB per billion parameters, plus a couple of GB of overhead. An 8B model fits on a laptop; a 32B model wants a 24 GB GPU; a 70B model wants 48 GB or two GPUs. Nothing about this requires the newest data-center hardware — which is precisely the point, and why the ecosystem keeps investing in engines that run well beyond CUDA: on Apple Silicon, AMD, and CPUs.
When self-hosting wins — and when the API wins
Self-hosting is not automatically cheaper or better. It wins on specific grounds:
- Privacy and data residency. The prompt never leaves your infrastructure. For healthcare, legal and finance workloads — or any enterprise with data-localization obligations — this is often the deciding argument, full stop.
- Cost at sustained volume. APIs price per token; a GPU prices per hour. Bursty, low-volume workloads favor the API. A steady high-volume pipeline (classification, extraction, summarization running all day) crosses over — the same GPU-hour serves millions of tokens, and vLLM-class batching is what makes the arithmetic work.
- Latency control and no rate limits. Your box, your queue.
- Model control. The provider can’t deprecate, quietly update or safety-tune your model out from under a validated workflow. For regulated systems that were evaluated against a specific model version, pinning weights locally is a compliance feature.
The API wins on frontier capability (the very best models are not open-weight), on zero ops burden, and on elasticity. In practice the strongest production pattern we see is routing: a self-hosted open model handles the routine 80% of requests, and the hard 20% escalates to a frontier API — the same buy-vs-build judgment we described in custom AI vs off-the-shelf, applied per-request.
A realistic starting path
- Prototype on Ollama on a dev machine. Prove the model quality on your task with your real data — an afternoon’s work.
- Evaluate honestly: run a fixed set of real inputs through the local model and the API model, compare outputs blind, and count. Feelings lie; counts don’t.
- Graduate to vLLM on a GPU server only when volume or privacy demands it — that’s the point where batching, monitoring and deployment discipline start to matter, and where MLOps stops being optional.
The open-source LLM stack rewards exactly the discipline it looks like it lets you skip. Treat a self-hosted model with the same evaluation rigor as any production system and it will quietly save you money for years; treat it as a toy and it will embarrass you in front of users.
Weighing self-hosted against API for a real workload? We’ll run the numbers with you — including the honest answer when the API is the right call.