Is It Time to Buy Your Own GPU for LLMs? The Honest Hardware Math

GPU prices are up, availability is tight, and every team running LLMs eventually asks the same question: should we just buy one? The answer isn’t ideological — it’s arithmetic, plus a few variables the arithmetic can’t capture. Here’s the full picture.

Buy, rent, or just call the API

There are three ways to run an LLM, and they sit on a clean spectrum of commitment:

• API (pay per token). You send a prompt, get a response, pay for the tokens consumed. Zero hardware, zero ops. OpenAI, Anthropic, Google and others price frontier models in the range of $1–15 per million input tokens and $2–60 per million output tokens, depending on model tier.
• Cloud GPU (pay per hour). You rent a machine with a GPU attached — an A100 for roughly $1.50–2.50/hr, an H100 for $2.50–4.00/hr — install a serving engine, and run whatever model you want. You own the ops; they own the hardware.
• Owned GPU (pay once, run forever). You buy the card, put it in a machine, and amortize the cost over its useful life. Ongoing cost is electricity, cooling and your time.

The right tier depends on exactly two things: how many tokens you burn per day, and whether the data can leave your building. Everything else is commentary.

The break-even math

Let’s make this concrete. Take a common setup: an RTX 4090 (24 GB VRAM, street price around $1,800–2,200) running Llama 3 8B at 4-bit quantization through vLLM or llama.cpp.

That card will generate roughly 80–120 tokens per second for a single stream, and with continuous batching under concurrent load, aggregate throughput can reach several hundred tokens per second. Call it an effective 200 tokens/sec on a sustained mixed workload, or roughly 17 million tokens per day running around the clock.

Now compare to a mid-tier API — say $0.50 per million input tokens and $1.50 per million output tokens, blended to roughly $1/million tokens for a model of comparable 8B-class quality. At 17 million tokens per day, that’s $17/day via API, or about $510/month.

An owned 4090 setup costs roughly:

• GPU: ~$2,000 (one-time)
• Host machine: ~$800–1,500 (CPU, RAM, PSU, case — one-time)
• Electricity: a 4090 pulls 350–450W under sustained load. At US average rates (~$0.14/kWh), that’s roughly $35–45/month running 24/7
• Amortized over 2 years: hardware cost alone works out to ~$115–145/month

Total monthly cost of ownership: roughly $150–190/month to serve the same 17 million daily tokens that would cost $510/month via API.

The crossover point is somewhere around 5–6 million tokens per day — below that, the API is cheaper when you factor in the zero-ops convenience. Above it, owned hardware pulls ahead fast, and the gap widens with every additional token.

Two caveats that shift the line:

1. Frontier models change the math entirely. The comparison above uses an 8B open model against a comparably-sized API model. If your task genuinely needs a frontier 400B+ model, you can’t run it on a single consumer GPU — and the API is the only practical option until you’re ready for a multi-GPU server.
2. Utilization is everything. A GPU sitting idle overnight is amortizing against zero tokens. The math only works at sustained volume — if your workload is bursty, cloud GPUs you can spin down beat owned hardware you can’t.

Which GPUs actually matter for LLMs

LLM inference is memory-bandwidth-bound, not compute-bound. The model’s weights sit in VRAM, and every generated token requires reading through them. This means the single most important spec is how much VRAM the card has and how fast it can read from it. Clock speed and CUDA core count are secondary.

The practical landscape:

• RTX 4090 (24 GB, ~$2,000) — the workhorse of local inference. Enough VRAM for any model up to ~32B parameters at 4-bit quantization. Excellent memory bandwidth (1 TB/s). The best performance-per-dollar for serious local work today.
• RTX 5090 (32 GB, ~$2,000–2,500) — the next generation. More VRAM means comfortably fitting 32B models with room for larger context, and the bandwidth improvement pushes tokens-per-second further. Worth waiting for if you haven’t bought yet.
• RTX A6000 / RTX 6000 Ada (48 GB, ~$4,500–6,500) — prosumer cards built for sustained workloads. 48 GB fits 70B models at 4-bit. No consumer power-limit throttling, ECC memory, designed to run 24/7. The tax for “professional” branding is real, but so is the reliability.
• A100 (40 or 80 GB, ~$8,000–15,000 used) — the data-center standard of the previous generation. Still excellent, widely available on cloud rental, and the 80 GB variant comfortably serves 70B models. Buying one used is a legitimate strategy if you know what you’re getting.
• H100 / H200 (80–141 GB, $25,000–40,000) — current-generation data-center GPUs. Transformative throughput under concurrent load, but you’re buying a server at this point, not a card. The economics only make sense at serious scale or when latency requirements are extreme.
• Apple Silicon (M2/M3/M4 Ultra, 64–192 GB unified memory) — a genuine dark horse. Unified memory means the model doesn’t need to fit in a discrete GPU’s VRAM — it shares the system’s full memory pool. Bandwidth is lower than a dedicated GPU, so tokens-per-second suffers, but for development, prototyping and low-concurrency private use, a Mac Studio running Ollama with a 70B model is a remarkably practical setup. No driver headaches, near-silent, sips power.

What fits in how much VRAM

At 4-bit quantization (the standard quality-to-size sweet spot — here’s why), the rule of thumb is ~0.6 GB per billion parameters, plus 1–2 GB overhead for the KV cache and runtime:

Two things this table hides:

• Context length eats VRAM too. The KV cache grows with context size. A 70B model at 4-bit fits in 48 GB at 4K context — but at 32K context, the KV cache alone can consume another 10–15 GB, pushing you into the next hardware tier.
• Concurrent users multiply memory. Each simultaneous request needs its own KV cache state. Serving one user and serving twenty on the same model are very different VRAM problems — this is where vLLM’s PagedAttention earns its keep, but even with it, more users means more memory.

The costs nobody talks about

The GPU price tag is the easy part. Here’s what comes after:

• Power. A 4090 under sustained inference load draws 350–450W. An H100 system pulls 700W+ from the GPU alone, before you count the host. Running 24/7, a single high-end GPU adds $35–80/month to your electricity bill — more in regions with expensive power. This is also a heat problem: that wattage becomes heat in your server room (or your office — we’ve seen it).
• Cooling and noise. Consumer GPUs with triple-fan coolers are genuinely loud under sustained load. A 4090 in a workstation under your desk will let you know it’s working. Data-center GPUs assume rack cooling and are even louder. If you’re running in an office, factor in the cost of moving the machine somewhere tolerable.
• Depreciation. GPU generations arrive roughly every two years, and each generation typically doubles inference throughput per dollar. A 4090 bought today will still work in 2028, but a 5090 will likely do the same job at half the power and twice the speed. Amortize over two years, not five — that’s the realistic useful life as a competitive inference platform.
• Ops burden. Driver updates, CUDA version management, model updates, monitoring, restarts after crashes. None of this is hard, but it’s not zero — and it’s time your team isn’t spending on the product. For a single developer box this is trivial; for a production-facing server it’s a real ops line item.
• Opportunity cost of capital. $2,000–15,000 locked in depreciating hardware is $2,000–15,000 not spent on engineering time, cloud credits you can turn off, or the API calls that would have gotten you to market three months earlier.

Cloud GPUs: the middle path

Renting a GPU by the hour gives you owned-hardware performance without the commitment. The practical landscape:

• Major clouds (AWS, GCP, Azure) — A100 and H100 instances, $1.50–4.00/hr, with all the usual cloud infrastructure around them (networking, storage, monitoring). Reliable, expensive, and overkill for a single-model inference job unless you’re already in that ecosystem.
• GPU-specialized providers (Lambda, RunPod, Vast.ai, CoreWeave) — often 30–60% cheaper than major clouds for raw GPU hours, with less surrounding infrastructure. RunPod and Vast.ai offer spot-style pricing under $1/hr for A100s, making short experiments remarkably cheap.
• Serverless GPU (Modal, Replicate, Together) — pay only for active inference seconds, scale to zero when idle. The economics are compelling for bursty workloads: a model that serves 50 requests in a burst and then sleeps for an hour costs almost nothing in idle time, versus an always-on instance burning $2/hr to wait.

When renting beats owning:

• You’re experimenting with multiple model sizes and don’t know what you need yet
• Your workload is bursty — heavy during business hours, dead overnight
• You need a GPU tier you can’t justify buying (an H100 for a quarterly fine-tuning run)
• You want to avoid ops entirely and treat GPU compute like a utility

When owning beats renting:

• Sustained 18+ hours/day utilization — the hourly cost overtakes amortized ownership fast
• Data can’t leave your premises — there’s no cloud that satisfies that constraint
• You need predictable, fixed costs — GPU cloud pricing is volatile and can spike during shortages

The decision framework

Skip the philosophy; here’s the checklist:

1. Can your task use an 8B-class model effectively? Test this first — run your actual prompts through Llama 3 8B via Ollama and compare against the API model you’re using now. If quality holds, the entire GPU conversation becomes affordable. If you genuinely need 70B+, your hardware budget just quintupled.

2. How many tokens per day, sustained? Under 2 million/day → API, don’t overthink it. 2–5 million/day → run the numbers, cloud GPU might be the sweet spot. Over 5 million/day sustained → owned hardware starts winning clearly.

3. Does the data need to stay on-premises? If yes, that overrides the cost math — you’re buying or renting private infrastructure regardless. For regulated industries (healthcare, finance, legal), this is often the deciding factor before any arithmetic happens.

4. Do you have someone to maintain it? A GPU under a desk running Ollama is trivial to maintain. A production vLLM server with uptime requirements, monitoring and failover is an ops role. Be honest about which one you’re signing up for.

5. Is your workload bursty or sustained? Bursty → serverless GPU or cloud instances you can stop. Sustained → owned hardware. The worst economic outcome is buying a $2,000 GPU that runs 4 hours a day — you’re paying owned-hardware prices for cloud-GPU utilization.

The honest summary

The GPU is not the hard part — the decision is. For sustained, high-volume inference on open models where data privacy matters, buying a GPU pays for itself in months, not years. For everything else — bursty workloads, frontier models, small teams without ops capacity — the API or a cloud GPU is the right call, and spending $2,000 on a card that sits idle most of the day is the most expensive option of all.

The right answer is almost always boring: start with the API, measure your real token volume for a month, and buy hardware only when the math is unambiguous. The software stack will be ready when you are.

Need help sizing the infrastructure for an LLM workload? That’s a Tuesday for us — let’s run the numbers together.