Core ML Inference Performance in 2026: Benchmarks, Latency, and What the Numbers Mean

The measurement problem

Core ML inference latency is not a single number. It is a distribution shaped by four independent variables: the hardware, the compute unit targeted, the precision of the model weights, and the thermal state of the device at inference time.

Benchmark articles that report a single figure — “8ms inference” — are describing one point in that space. The number is not wrong, but it is incomplete. Understanding the full distribution is what separates a benchmark from an architecture input.

The constraint that shapes every production decision: you cannot control which device your user runs, and you cannot control thermal state. You can control compute unit selection, model precision, and when you schedule inference.

What Core ML actually runs on

Apple Silicon devices contain three distinct compute resources that Core ML can target: the CPU, the GPU, and the Neural Engine (ANE). Each has a different performance profile and a different power cost.

Latency benchmarks: 2026 reference points

The figures below reflect measurements on current-generation Apple Silicon. They are representative ranges, not guarantees — device condition, thermal state, and model specifics all shift the numbers.

Classification models

Image classification models converted to Core ML with INT8 weight quantization, running on A17 Pro or M3 hardware:

Text classification models (BERT-Mini, DistilBERT) with 4-bit quantization:

Generative and language models

Autoregressive generation latency is measured in tokens per second, not total latency — first-token latency and sustained throughput are separate figures. For a 3B parameter model quantized to 4-bit:

First-token latency under 100ms is imperceptible. Above 300ms, users perceive a pause.

Cloud API round-trip comparison

The on-device numbers above compare against cloud API round-trips of 200–800ms under normal network conditions — and that range assumes a reliable connection. On mobile networks, p95 latency for a cloud inference call frequently exceeds 1 second.

On-device inference is not faster in every scenario. For very large models — 70B+ parameters — cloud inference is still faster in absolute terms. But for models in the 3B–7B range with quantization, on-device inference on current Apple Silicon is competitive with cloud latency on a good connection, and strictly faster on a degraded one.

At 3NSOFTS, on-device AI integration targets sub-10ms latency for classification and feature extraction tasks, and under 150ms first-token for generative tasks — figures that hold without any network dependency. Full Core ML vs cloud API comparison →

The variables that move the numbers

Compute unit selection

The default .all configuration works well for models that were converted cleanly and contain only ANE-supported operations. When a model contains unsupported ops, Core ML silently routes those layers to the CPU — and total latency reflects the CPU bottleneck, not ANE throughput.

The diagnostic: run MLModel with .cpuAndNeuralEngine and compare against .all. If latency is similar, the model is already running on the ANE. If .cpuAndNeuralEngine is significantly slower, the GPU was doing meaningful work. If both are slow, the model has CPU-bound operations.

Model precision and quantization

FP32 weights are the starting point. FP16 halves the memory footprint with negligible accuracy loss for most tasks. INT8 halves it again. 4-bit quantization — now well-supported via Core ML Tools 8.x — reduces a 3B parameter model to approximately 1.5GB, which fits comfortably within the memory budget of an iPhone 15 Pro.

The latency improvement from FP32 to INT8 on the ANE is typically 30–50% for classification models. For generative models, quantization affects sustained throughput more than first-token latency. Core ML optimization techniques deep-dive →

Batch size and input shape

Core ML models are compiled for a fixed input shape by default. Sending a batch of 8 images through a model compiled for batch size 1 forces sequential inference — you get 8x the single-image latency, not a batched speedup.

If your use case involves batch inference, compile the model with a flexible input shape using MLModelConfiguration and MLMultiArray with the appropriate batch dimension. The ANE handles batched operations efficiently when the model is compiled to expect them.

Thermal state and battery pressure

This is the variable most benchmarks ignore. Apple Silicon chips throttle aggressively under sustained thermal load. A model that runs in 8ms on a cold device may run in 18–25ms after 10 minutes of continuous inference.

ProcessInfo.thermalState surfaces the current thermal state. Production architectures that run inference in a loop — real-time classification, continuous audio processing — need to check thermal state and reduce inference frequency under pressure. This is not optional. It is the difference between an app that works in a demo and one that works in production.

MLComputeUnits: what each option actually does

.cpuAndNeuralEngine is underused. For models that are fully ANE-compatible, it produces the best combination of latency and power draw — the GPU is not loaded, and the ANE handles the full inference path.

Measuring inference in production

Xcode Instruments has a Core ML template. Use it. It shows per-layer execution time, compute unit routing, and memory allocation — the three things you need to diagnose a latency problem.

For production monitoring, the minimal measurement pattern is:

let start = CFAbsoluteTimeGetCurrent()
let output = try model.prediction(input: input)
let latency = CFAbsoluteTimeGetCurrent() - start
// Log latency alongside ProcessInfo.thermalState

Log thermal state alongside latency. Without it, you cannot distinguish a model performance regression from a thermal throttling event.

The p95 figure is what matters for user experience, not the mean. A model with a 5ms mean and a 200ms p95 will produce visible stutters. Measure the distribution, not the average.

Apple Foundation Models and the 2026 baseline

Apple Foundation Models, introduced with Apple Intelligence, run entirely on-device via the ANE. They surface through the FoundationModels framework with a structured API — not as raw Core ML models.

Foundation Models tasks are fast enough for interactive use, but not fast enough for real-time inference in a tight loop. They belong in response to user actions, not in a 60fps render loop.

Foundation Models vs Core ML: when to use each →

What the numbers mean for architecture decisions

The latency figures above are inputs to architecture decisions, not outputs. The question is not “how fast is Core ML?” — it is “which inference path fits the latency budget of this specific feature?”

FAQs

Related articles