This is the framing that distinguishes Apple’s AFM third generation<\/a>, Google’s Gemini Nano on Android<\/a>, and Qualcomm’s AI Hub<\/a> from cloud-only assistants. All three assume a hybrid future. None of them pretend the cloud is going away.<\/p>\n\n\n\n The number that matters is the percentage of consumer AI tasks that should default to on-device<\/strong>. For 2026, that number is approximately 60% \u2014 based on the empirical distribution of tasks users actually run: rewriting, summarizing, formatting, intent classification, entity extraction, translation, image captioning, structured Q&A. For those tasks, the latency win is real (sub-50ms time-to-first-token vs 200-500ms round-trip), the privacy win is real (no data leaves the device for inference), and the cost win is real (zero per-query inference cost). Cloud still wins on broad-knowledge Q&A, complex multi-step reasoning, agentic tool use, and long-context tasks.<\/p>\n\n\n\n Apple’s third-generation Apple Foundation Models<\/a>, published June 8, 2026, are five distinct models:<\/p>\n\n\n\n The 3B is the workhorse. It handles summarization, rewriting, intent classification, and structured Q&A on-device with no network round-trip. The 20B Core Advanced is the new thing \u2014 a sparsely-activated model that fits in the memory budget of a high-end consumer device while delivering capabilities closer to a 70B dense model on specific benchmarks. The technical details \u2014 training corpus (DCLM-7B-derived), architecture specifics, and evaluation methodology \u2014 are in the Apple Intelligence Foundation Language Models tech report<\/a>.<\/p>\n\n\n\n Developer access comes via the Foundation Models framework<\/strong>. For teams already running local LLM inference on Mac silicon<\/a>, the Foundation Models framework is a path off llama.cpp and onto Apple’s first-party runtime with no per-query rate limits., a unified Swift API. No API keys, no rate limits, no per-query billing. You instantiate a Google’s on-device path on Android is Gemini Nano<\/a>, and unlike Apple’s model family, Nano runs in a dedicated system service called AICore<\/strong>. AICore is sandboxed: it has no direct internet access, can only download model updates through Private Compute Services, and isolates every request so inputs and outputs are not retained after processing.<\/p>\n\n\n\n The Pixel 9 Pro runs Gemini Nano v2 \u2014 note: not v3, which requires newer hardware for the latest Gemini Intelligence features. On the Pixel 9 Pro, Nano v2 hits approximately 510 tokens\/sec for prefix-input processing (think: prompt ingestion, not generation) and roughly 11 tokens\/sec for text-to-text generation. That’s the upper bound for current shipping hardware; the iPhone’s 30 tokens\/sec on AFM 3 Core is meaningfully faster on the generation side because Apple’s ANE is purpose-built for transformer inference.<\/p>\n\n\n\n The interesting engineering move from Google in 2026 is frozen Multi-Token Prediction<\/a> \u2014 retrofitting an MTP head onto a frozen base model without retraining the underlying weights. The MTP layer predicts multiple tokens in parallel and rejects inconsistent predictions during decoding. Result: 50%+ speedups on specific tasks (summarization, proofreading) without the cost of a full retrain. This is the playbook for shipping inference improvements to a billion-device fleet without touching the base model \u2014 and it’s a signal that Google expects Nano to be a multi-year on-device platform, not a one-shot release. The same small-model-vs-large-model tradeoff that powers Nano v2 is what we covered in Small language models in 2026: when 7B beats 70B<\/a>.<\/p>\n\n\n\n One implication for users: the September 4, 2026 shutdown of Google Assistant on Pixel 9 and Galaxy S25 devices means those phones are now permanently locked to a slower Gemini path. Pixel 9 Pro users get the full Nano v2 experience; older Pixel 9 \/ S25 users get a less capable cloud-routed fallback. If you’re shopping for a phone specifically for on-device AI, the generation of the silicon matters more than the marketing copy.<\/p>\n\n\n\n Qualcomm’s path is the most developer-centric of the three \u2014 partly because they’re not trying to own the OS, they’re selling the silicon and the runtime. The two pieces you actually need:<\/p>\n\n\n\n The workflow in 2026 looks like this: you prototype a model in PyTorch, submit a compile job to the AI Hub targeting a Snapdragon device, download the compiled binary, then either bundle it in your app or serve it from a GenieX-compatible backend. The Python install is The friction in practice is the model-zoo curation. Qualcomm’s AI Hub hosts a growing catalog of pre-compiled models (Qwen3-4B, Llama variants, several domain-specific fine-tunes), but anything not in the catalog requires you to do the compile-job round-trip yourself. For a small team shipping one model on two device classes, this is a 2-3 day lift. For a team that needs to support eight device classes with custom fine-tunes, it’s a 2-3 week investment that compounds. The broader silicon landscape \u2014 NVIDIA Rubin, Groq 3, Cerebras \u2014 sits in a separate competitive track; our AI Hardware Race 2026 analysis<\/a> covers the data-center side, while this post is the device side of the same story.<\/p>\n\n\n\n Qualcomm’s Snapdragon gets the developer mindshare, but roughly 40% of premium Android phones ship with MediaTek silicon \u2014 Oppo Find X8, Vivo X200, Realme GT series. The Dimensity 9400 (and its refresh 9400+) ship with the NPU 890, MediaTek’s 8th-generation neural processor.<\/p>\n\n\n\n The Dimensity 9400 launch announcement<\/a> claimed 80% faster LLM prompt performance and 100% faster diffusion generation compared to the Dimensity 9300, plus 35% greater power efficiency. The AI Benchmark score of 6,773<\/a> (vs 4,911 for the 9300) confirms the headline numbers \u2014 a 38% year-over-year NPU score uplift.<\/p>\n\n\n\n Two things make MediaTek’s silicon interesting beyond raw performance. First: FP8 inference support<\/strong>. Most on-device runtimes are still on INT4 \/ INT8; FP8 unlocks a denser model class with reasonable accuracy retention. Second: on-device LoRA training<\/strong>. The NPU 890 can fine-tune LoRA adapters locally, which means personalization happens on-device without sending user data to a training server. Mixture of Experts (MoE) inference is also supported \u2014 important because the next wave of small models (under 7B) is increasingly MoE. The 35% power-efficiency uplift on NPU 890 also feeds back into the AI Energy Consumption 2026<\/a> conversation: when the per-query watt-hour drops, the case for on-device inference at scale gets stronger.<\/p>\n\n\n\n The friction: the developer story is more fragmented than Qualcomm’s. NeuroPilot is MediaTek’s SDK but the ecosystem of pre-compiled models is smaller. The MediaTek AI Hub equivalents<\/a> exist but lag Qualcomm’s catalog in breadth. If you’re shipping a model that needs to run on both Snapdragon and Dimensity devices, plan for two compile pipelines.<\/p>\n\n\n\n On the laptop side, Intel’s Core Ultra processors (Meteor Lake, Lunar Lake, and the 2026 Panther Lake refresh) include a discrete NPU \u2014 typically 11-13 TOPS at INT8 on current shipping silicon. The honest assessment of laptop NPUs for LLM inference comes from the NITRO paper<\/a> by Anthony Fei and Mohamed S. Abdelfattah (Intel, December 2024, still the canonical reference): running LLMs on Intel NPUs is technically possible but throughput-constrained<\/strong>. Models larger than 3B parameters face severe memory bottlenecks compared to GPU-accelerated llama.cpp running on the integrated Arc GPU.<\/p>\n\n\n\n The working path on Intel hardware is OpenVINO GenAI with the Intel did achieve first and only full NPU support in MLPerf Client v0.6<\/a> (May 2025), which matters for benchmark compliance and for product teams who need auditable NPU utilization numbers for marketing claims. But the daily reality for an LLM app developer in 2026 is that the laptop NPU is best for always-on background tasks (noise suppression, eye tracking, onlooker detection) rather than as a primary LLM inference path. For real LLM work on Intel laptops, the iGPU is usually the better target. The 32K context window story \u2014 what fits locally, what has to be summarized or compressed before the model can use it \u2014 is in LLM Context Windows: Why Your 1M-Token Model Only Uses 32K<\/a>.<\/p>\n\n\n\n Two different privacy models, both legitimate, with different trade-offs.<\/p>\n\n\n\n Google’s AICore<\/strong> runs Gemini Nano fully on-device with no network connection at all<\/a>. The system service is sandboxed; model downloads happen through Private Compute Services (a separate, more restricted sandbox); every request is isolated from the next; inputs and outputs are not retained after processing. If the local model can’t satisfy the request, Android escalates to a cloud Gemini endpoint \u2014 but that’s a separate API call with its own consent flow, not the Nano path. The privacy guarantee: nothing about the Nano inference leaves your device, period<\/strong>.<\/p>\n\n\n\n Apple’s Private Cloud Compute<\/strong> is hybrid. The on-device AFM 3 Core and AFM 3 Core Advanced handle simple tasks entirely locally. For complex tasks (agentic tool use, multi-step reasoning, image generation), Apple routes the request to PCC \u2014 stateless Apple Silicon servers where requests are encrypted, processed, not retained, and inaccessible even to Apple staff. The verification model is published: PCC nodes broadcast a transparency log of their software state, and you can cryptographically verify that the server your request landed on is running the publicly-disclosed software.<\/p>\n\n\n\n In June 2026, Apple expanded PCC to use Google Cloud with NVIDIA GPUs<\/a> for some agentic workloads. The Verge’s coverage of the announcement \u2014 “Apple says its AI is still private, even when it’s running on Google’s servers” \u2014 captures the architectural reality: the data confidentiality and non-retention guarantees are designed to hold across the underlying compute provider. Independent academic analysis<\/a> from TU Darmstadt researchers confirms the implementation holds the documented properties.<\/p>\n\n\n\n Pragmatically: if you need the strongest possible on-device guarantee (zero data egress, period), Google’s Nano path is the cleanest. If you need the strongest possible cloud-with-strong-guarantees path, Apple’s PCC is the cleanest. Both are legitimate, both are defensible, both have shipped at scale.<\/p>\n\n\n\n\n The number product teams argue about most is the accuracy gap between on-device small models and cloud frontier models. In 2026, the headline number is approximately 16.8 MMLU points<\/strong> \u2014 edge 8B models cluster around 65.2 MMLU; cloud 70B models cluster around 82.0 MMLU. That’s a real gap on a broad-knowledge benchmark.<\/p>\n\n\n\n But MMLU measures general knowledge across 57 academic subjects, which is not what most production tasks need. For specific production tasks \u2014 intent classification, entity extraction, structured formatting, summarization, rewriting \u2014 tuned 1B to 7B on-device models often match cloud counterparts within a few percentage points. The accuracy delta that matters in production is the delta on your<\/em> workload, on your<\/em> evaluation set, not the MMLU delta.<\/p>\n\n\n\n The pragmatic rule for 2026 product decisions: on-device for latency-sensitive, cost-sensitive, and privacy-sensitive paths; cloud for complex reasoning and broad general knowledge<\/strong>. The 5-7 point accuracy trade is usually worth the latency, cost, and privacy wins for the local path. For multi-document summarization, agentic tool use, code generation on unfamiliar codebases, or anything that requires broad factual recall, route to the cloud. We mapped the unit economics of the cloud-routed path in AI Inference Cost 2026: what one prompt actually costs<\/a> \u2014 the short version is that for sustained workloads, the local path’s per-query cost of zero starts to dominate any cloud optimization. The two paths complement; they don’t compete.<\/p>\n\n\n\n If you’re a developer in 2026 trying to ship on-device AI features, here’s what the SDK landscape actually looks like in practice.<\/p>\n\n\n\n Apple Foundation Models framework<\/strong>: Swift-native, no API keys, no rate limits, 32K context window. The most polished developer experience of the three \u2014 Apple’s WWDC26 sessions (sessions 241 and 319) walk through the entire API. Trade-off: iOS\/macOS only.<\/p>\n\n\n\n Android AICore + Gemini Nano<\/strong>: Kotlin\/Java API, integrates with the system service. Two Nano model variants available (v2 on Pixel 9 Pro, v3 on newer hardware). Google’s Android AI development documentation<\/a> covers the integration patterns. Trade-off: device coverage is uneven across Android OEMs, and not all Snapdragon\/Dimensity devices support the AICore runtime.<\/p>\n\n\n\n Qualcomm AI Hub + GenieX<\/strong>: Compile-job workflow on the Hub, GenieX runtime for deployment. Works on any Hexagon NPU device, which means most Snapdragon 8 Gen 2+ Android phones. The AI Hub docs<\/a> are detailed; the GenieX README<\/a> is pragmatic. Trade-off: model coverage depends on your willingness to run compile jobs; pre-compiled bundle catalog is smaller than Apple’s.<\/p>\n\n\n\n MediaTek Genio AI Hub + NeuroPilot<\/strong>: Equivalent to Qualcomm’s stack but smaller catalog. LiteRT \/ Generative AI support<\/a> is documented but the model zoo is still catching up. Trade-off: requires dual compile pipelines if you also support Snapdragon.<\/p>\n\n\n\nApple Foundation Model: the AFM 3 family (3B on-device, 20B sparsely activated, three cloud tiers)<\/h2>\n\n\n\n
\n
LanguageModelSession<\/code>, optionally pointing it at the PrivateCloudComputeLanguageModel<\/code> for tasks that need the cloud tier, and you get a 32K context window with built-in reasoning controls.<\/p>\n\n\n\nGemini Nano: AICore, frozen multi-token prediction, and the 22-tokens\/sec ceiling<\/h2>\n\n\n\n
Qualcomm AI Hub + GenieX: the developer SDK for Android phones<\/h2>\n\n\n\n
\n
pip install geniex<\/code>; running a model from Hugging Face is geniex infer unsloth\/Qwen3.5-2B-GGUF<\/code>. The Android SDK integrates the same runtime into a native application.<\/p>\n\n\n\nMediaTek Dimensity 9400: the other half of Android<\/h2>\n\n\n\n
Intel Core Ultra NPU: laptop-side AI acceleration<\/h2>\n\n\n\n
LLMPipeline<\/code> class, models compiled with static shape requirements and symmetric INT4 quantization (group size 128). Don’t try OVModelForCausalLM<\/code> \u2014 it fails on dynamic shapes. Use LLMPipeline<\/code>. Once that’s set up, the throughput is usable for short prompts (sub-1K tokens) but degrades on longer contexts.<\/p>\n\n\n\nPrivacy boundary: AICore’s sandbox vs Apple PCC’s stateless cloud<\/h2>\n\n\n\n
![\"Editorial](\"https:\/\/i0.wp.com\/aimade.tech\/wp-content\/uploads\/2026\/08\/on-device-vs-cloud-2026-chart.png?ssl=1\")
Accuracy vs cloud: the 16.8-point MMLU gap and when it matters<\/h2>\n\n\n\n
The developer build-out: SDKs, frameworks, and where the friction is<\/h2>\n\n\n\n