The Artificial Analysis mobile inference data shows a widening gap between what phones can theoretically run and what they can sustain. Small models won, decoding is slow, and thermals still rule everything.
Phones can now run models that would've made a data center laugh three years ago. The Artificial Analysis mobile hardware page tracks exactly how fast, how hot, and how usable that promise actually is. And the numbers are complicated in ways the marketing slides never mention.
If you've been assuming your iPhone can casually run a 7B model at ChatGPT-like speeds, the data will recalibrate you fast.
Phone AI inference benchmarks measure how many tokens per second a mobile device generates from a local language model, alongside prefill latency, memory ceiling, and thermal throttling behavior over time. The Artificial Analysis mobile stack tracks these across flagship SoCs from Apple, Qualcomm, and MediaTek using standardized model builds and quantization schemes.
That's the short answer. The longer answer is where it gets interesting.
Small models (roughly 1B to 4B parameters) dominate the phone category because RAM is the hard ceiling on everything else. A 3B model in 4-bit quantization sits around 1.8 GB in memory. A 7B model at the same precision needs closer to 4 GB, which starts fighting the OS, the browser, and every background app for headroom. So even when a phone technically has 8 GB or 12 GB of RAM, the model doesn't get to keep it.
A few things stand out from the current mobile inference numbers.
And no, none of this is close to cloud speeds. Not gonna lie.
Artificial Analysis runs the same quantized model builds across a fleet of devices, measuring four numbers that actually matter for user experience:
The methodology avoids app-level noise by going through native runtimes (MLC LLM, llama.cpp, MLX on Apple silicon). This matters more than most benchmark reports admit. App-shipped models often sit behind aggressive throttling for battery reasons, so what a user actually feels inside a shipped iOS or Android app is often 30 to 50% slower than what the benchmark put to work records.
The framework also splits cold-start from warm-run numbers, which is a big deal. A phone that hits 25 tokens/sec on the first prompt might sag to 12 tokens/sec three prompts in as the SoC downclocks to protect itself. Marketing loves the first number. Users experience the second.
Below is the rough shape of what the data shows across device tiers. Exact figures shift per model and quantization scheme, so consult the source dashboard for current numbers.
| Device tier | Example SoC | Typical decode (tok/s, 3B q4) | Notes |
|---|---|---|---|
| 2026 flagship | A18 Pro, SD 8 Elite Gen 5 | Well above reading pace | Sustained throughput limited by heat |
| 2024-25 flagship | A17 Pro, SD 8 Gen 3 | Comfortably interactive | Some throttling after two minutes |
| Mid-tier 2025 | Dimensity 8400 class | Marginal for chat | Fine for short completions |
| Older flagship | SD 8 Gen 2, A16 | Slow but usable | Best kept to sub-2B models |
The gap between "runs a demo" and "runs for 30 minutes without turning your phone into a hand warmer" is bigger than most product pages admit.
Some honest takeaways from staring at this data.
Small models won, at least for now. The 3B parameter class (Llama 3.2 3B, Phi-3.5 Mini, Qwen 2.5 3B, Gemma 2 2B) is the sweet spot for consumer devices. They fit, they run at usable speeds, and their capability is genuinely useful for classification, summarization, and structured extraction. Nobody is running Claude Opus 4.6 on a phone. That's fine. That's not what phones are for.
But raw MMLU or HumanEval scores from the desktop benchmarks translate poorly to on-device reality, and the broader benchmark saturation problem means even those desktop numbers are getting harder to interpret year over year. A 3B model that scores respectably on MMLU will still fumble multi-step reasoning that a 70B cloud model handles trivially. So the practical benchmark for on-device work is task-specific eval, not academic leaderboards. If you want to run your own local numbers, Homebench is a solid starting point for measuring tokens/sec and memory pressure across quantization schemes. If your app needs to categorize emails or extract JSON from user text, the small models are ready. If it needs to plan a trip or debug code, you're still calling an API.
Prefill matters more than decode for RAG. If you're building an app that stuffs 4K tokens of context into every call, prefill throughput is what determines whether the app feels snappy or sluggish. Decode speed dominates the perception only in chat-style streaming where the response is short and the user is watching it appear. Most retrieval-augmented apps aren't that.
Battery is the silent benchmark. None of the public leaderboards score watt-hours per 1M tokens generated, but on a phone that number decides whether users open your app twice. A model that runs at 20 tok/s but drops the battery 8% per session will get uninstalled. This is the number every product manager should be asking for, and almost nobody publishes it.
The old rule that four-bit quantization is a compromise is aging poorly. QAT variants are closing the quality gap fast.
A couple of findings ran against expectation.
The Qualcomm Hexagon NPU underperforms its spec sheet in most public numbers because tooling maturity lags Apple's MLX. On paper, Snapdragon 8 Elite Gen 5 has more raw NPU throughput than the A18 Pro. In practice, Apple's story is boring in comparison: MLX plus Metal just works, and the MLX-community model releases keep pace with new architectures within weeks of release. This is one of those cases where the closed vendor is winning on pure developer experience.
MediaTek's Dimensity 9400 series is punching above its price bracket too. Not gonna call it a win over Apple silicon, but the perf-per-dollar has genuinely narrowed enough that Android OEMs shipping on Dimensity now have a credible AI story.
Also worth noting: quantization-aware training is showing up in more model releases. Google's Gemma 3 QAT variants are a good example, where the quality gap between q4 and fp16 keeps shrinking. The old assumption that four-bit quantization is a meaningful quality tax is aging poorly, especially for chat and completion tasks.
And the llama.cpp mobile discussion threads keep surfacing performance regressions and wins that never make it into vendor benchmarks. That community-driven signal is often more reliable than official numbers.
Practical guidance, based on what the benchmarks actually say and not what the keynote demos promise:
If you want the raw comparisons, Artificial Analysis publishes them openly and updates them frequently. The MLC LLM project also runs an independent leaderboard with historical data across a wider set of Android devices.
The bigger picture: on-device inference has moved from "cute demo" to "actually shippable for narrow tasks." It's still nowhere close to displacing cloud APIs for anything requiring real reasoning, long context, or agentic workflows. But for the 80% of AI features that are classification, extraction, transcription cleanup, or short generation, the phone is now a legitimate deployment target. That's a genuine shift, and it should change what a lot of apps look like architecturally. Especially the ones that were paying API fees to do something a 3B model could handle for free, on the device the user already owned.
Technically yes at 4-bit quantization (roughly 4 GB), but you'll fight the OS for RAM and hit thermal throttling within a few minutes of active use. Most developers shipping to iOS target the 3B range with MLX for a reason: it fits comfortably in memory, sustains speed under load, and leaves room for the rest of your app to function.
MLC LLM currently leads on raw throughput for most Snapdragon and Dimensity chips because it compiles models directly to the target device's GPU and NPU. llama.cpp is more portable and easier to integrate, but typically runs 20 to 40% slower than MLC on the same hardware. For production Android apps, MLC is worth the integration effort if speed matters.
Public numbers are scarce, but community measurements suggest a 3B model at sustained generation drains roughly 5 to 10% of a typical flagship battery per 10-minute session, depending on quantization and SoC. This is why most mobile AI features run in short bursts rather than continuous chat, and why background inference is generally a bad idea.
For anything below 4-bit, absolutely yes. QAT variants like Google's Gemma 3 QAT builds show substantially less quality degradation than post-training quantization at the same precision. If you're targeting q3 or below to fit on mid-tier devices, QAT is often the difference between a shippable feature and one that hallucinates constantly.
Use on-device for privacy-sensitive tasks, offline scenarios, short-context classification and extraction, and any high-volume feature where per-call API costs would kill margins. Use cloud APIs for long context, complex reasoning, agentic workflows, and anything a user would tolerate a 500ms round trip for. Hybrid architectures (local for cheap fast-path, cloud for hard cases) are becoming the default for serious mobile AI apps.
