“On-device AI” gets discussed as if the only question is whether a model fits in RAM. It isn’t. An iPhone 17 Pro ships with 12 GB of memory, and you will never get close to using it for a language model — the operating system decides that long before your model does.
This post is the practical version of that constraint: what actually fits, how fast it actually runs, and where the boundary sits between what belongs on the phone and what still has to leave it. It’s the phone tier of the local-first router pattern, examined properly.
The number that actually constrains you
iOS kills apps that use too much memory. The mechanism is Jetsam, and the budget it enforces is not your device’s RAM.
On current hardware the per-app ceiling lands at roughly 6 GB, and it stays near 6 GB even on a 12 GB iPhone 17 Pro — going from 8 GB to 12 GB of physical RAM did not double what a single app may hold. In practice the kill threshold moves with device state, landing somewhere around 50–67% of total RAM depending on what else the system is doing.
So the honest budget looks like this:
- Your model weights
- The KV cache, which grows with context length
- Your entire actual app — UI, images, caches, everything a user expects to still be there when they come back
That last line is the one teams forget. A model that fits in 5 GB on a bench test leaves you almost nothing for the product wrapped around it, and the failure mode isn’t a slowdown — it’s the OS terminating your app mid-sentence. Budget the model at a fraction of the ceiling, not near it.
Two doors, and only one of them charges you rent
There are two fundamentally different ways to run a language model on an iPhone, and they have completely different cost structures.
Door one: Apple’s system model. The Foundation Models framework exposes a roughly 3B-parameter on-device model, compressed with 2-bit quantization-aware training and using KV-cache sharing between its two halves to cut memory use by about 37.5%. The important part for you isn’t the architecture — it’s the accounting. The model and its inference resources are managed centrally by the OS and shared across every Apple Intelligence feature, so calling it adds very little to your app’s own memory footprint.
That is a remarkable deal: a capable 3B model that costs you almost nothing against the ceiling that would otherwise kill you. The trade is that you get Apple’s model, Apple’s tokenizer, Apple’s guardrails and Apple’s update schedule. You cannot fine-tune it on your domain.
Door two: your own model, via Core ML on the Neural Engine or MLX on the GPU. Full control — your weights, your fine-tune, your quantization, your behaviour. And every single byte counts against your ~6 GB budget, along with the KV cache and the rest of your app.
The decision is usually straightforward: if a general-purpose 3B model does the job, take door one. Ship your own model when the task genuinely needs your data or your behaviour — a domain vocabulary, a specific output contract, a fine-tune that earns its keep. Paying rent on the memory ceiling for a model that performs the same as the free one is the most expensive mistake available here.
What actually fits, and how fast it really goes
Here are real measured figures from an iPhone 17 Pro (A19 Pro, 12 GB) running 4-bit quantized models, generation speed at a medium prompt:
| Model | On-disk size | Quant | Tokens/sec |
|---|---|---|---|
| Qwen3 0.6B | 351 MB | 4-bit | 62.2 |
| LFM2.5 1.2B | 663 MB | 4-bit | 59.7 |
| Llama 3.2 1B | 713 MB | 4-bit | 58.1 |
| Qwen3 1.7B | 984 MB | 4-bit | 39.5 |
| Gemma 3 1B (QAT) | 772 MB | 4-bit | 37.1 |
| LFM2.5 1.2B | 951 MB | 6-bit | 45.4 |
Four things worth pulling out of that table:
The sizes are small. A capable 1B model is a ~700 MB file. This is not a memory problem in the way people assume — the weights are a rounding error against a 6 GB ceiling. What grows dangerously is the KV cache at long context, which is the same dynamic that eats VRAM on a desktop GPU, just against a much tighter budget.
Sub-2B is genuinely usable. 58–62 tokens/sec is faster than a person reads. For classification, extraction, rewriting and short structured answers, this is not a compromised experience — it’s instant, with no network round trip at all.
Quantization is a real lever. The same 1.2B model at 6-bit instead of 4-bit costs 21–27% throughput and about 290 MB more. Unless your evals show 4-bit degrading the specific task, 4-bit is the sweet spot on a phone as it is elsewhere.
Parameter count doesn’t cleanly predict speed. Gemma 3 1B lands at 37.1 tok/s while Llama 3.2 1B hits 58.1 — same nominal size, 57% faster. Architecture and how well a model maps onto Apple silicon matter as much as size. Benchmark the specific model; don’t reason from the parameter count.
The Neural Engine isn’t where the gains went
This is the most counter-intuitive thing in current Apple silicon, and it changes how you should target the hardware.
The A19 Pro’s headline AI feature is a 16-core Neural Engine. But Apple also put dedicated Neural Accelerators inside every GPU core, and that’s where the generational leap landed. Measured across on-device inference workloads, the iPhone 17 Pro’s GPU runs 2.5–3.1× faster than the iPhone 16 Pro’s. The Neural Engine improved by 1–1.15× over the same generation — effectively flat.
The practical reading: if you are shipping your own model and you want the headroom this generation added, the GPU path via Metal and MLX is where it is. The Neural Engine remains excellent at what it was built for — sustained, power-efficient, fixed-function inference, which is exactly why Apple runs its own system model there — but it is not where the new performance lives.
Two consequences worth designing around. Apple’s system model runs on the ANE, so it doesn’t contend with your GPU work. And an iPad with an M-series chip runs the same models 1.2–2.2× faster than the iPhone, with the gap widening at longer contexts because memory bandwidth, not compute, is the binding constraint. If your product has an iPad tier, that’s real headroom you can spend on a bigger model.
The benchmark you can’t see
Every published tokens-per-second figure is a burst measurement. Phones do not sustain burst performance, and a benchmark run on a cold device tells you very little about minute five.
The measured drift is real but modest: Qwen3 0.6B starts at 63 tok/s and settles to 57 by the tenth consecutive run — about a 10% decline as heat builds. Smaller models stay flat, because they never generate enough load to trigger throttling in the first place.
That shape is the design guidance. Short, bursty inference is what phones are actually good at. A model that answers in two seconds and then goes quiet stays fast indefinitely. A model driving a continuous agent loop for ten minutes will get slower, get hot in the user’s hand, and drain battery in a way they will notice and blame you for — and none of that appears in a benchmark table. If your workload is genuinely sustained, that’s a signal it belongs somewhere other than the phone.
Designing for the real envelope
What all of the above adds up to, as rules:
- Try the system model first. If a 3B general model handles the task, you get capability without paying memory rent. Ship your own only when your data or behaviour genuinely requires it.
- Budget memory for the app, not the benchmark. Target a fraction of the ~6 GB ceiling. Account for the KV cache growing with context, and cap context deliberately rather than discovering the limit through crash reports.
- Prefer 4-bit. Move to 6-bit only when an eval set shows a quality loss that matters, and know you’re paying roughly a quarter of your throughput for it.
- Benchmark the specific model on the specific device. Same-size models differ by 57%. There is no substitute for measuring.
- Design for bursts. Short interactions stay fast forever; long loops throttle. Structure the feature around how the hardware actually behaves.
- Keep an escalation path. On-device handles the routine; hard cases go to a server. That’s the router pattern, and on a phone it’s not an optimization — it’s what makes the feature possible at all.
What still goes to the cloud
Being honest about the boundary is what makes the on-device part trustworthy:
- Long-horizon reasoning and planning. The gap between a 1–3B model and a frontier model is widest exactly here, and it doesn’t close with better prompting.
- Broad world knowledge. A 700 MB file cannot hold what a 400B model holds. Retrieval helps; it doesn’t erase the difference.
- Long-context work. The KV cache is the memory constraint, and a phone runs out well before a server does.
- Anything sustained. Thermals make a ten-minute local loop a worse experience than a one-second network call.
What stays on the device: classification, extraction, rewriting, tool selection, short structured generation, and anything touching data that shouldn’t leave — health, financial, personal. That last category is where on-device stops being an optimization and becomes the product. An app that works on a plane, keeps sensitive data on the phone, and carries no per-user inference bill is a fundamentally different proposition from one that doesn’t.
The bottom line
An iPhone in 2026 will comfortably run a 1–2B model at ~60 tokens/second in under a gigabyte, or give you a 3B system model for almost no memory cost at all. That is genuinely a lot — enough that most routine language work in a mobile product no longer needs a server.
What it will not give you is a frontier model in your pocket, an unlimited memory budget, or sustained throughput under load. The teams that ship good on-device AI are the ones that design for that envelope deliberately, instead of discovering it in production. The constraints are knowable — measure them first, and build the feature that fits.
Building something that has to run on the device itself? That’s what we do.