ONNX Runtime on iOS: Configure Core ML and Diagnose CPU Fallback

Quick answer

Use an ONNX Runtime build that includes the Core ML execution provider, register it before creating the session, and compare it with a CPU baseline using identical inputs. Then inspect graph assignment and device behavior. Registering a provider is not proof that the entire graph runs through it, or that Core ML schedules that work on the Neural Engine.

There are two distinct questions: which ONNX operations are delegated to Core ML, and which hardware Core ML uses for that delegated work? Keep the evidence for those questions separate.

This guide is for an app already using ONNX. If you are still choosing a runtime, start with Core ML vs ONNX Runtime for iOS.

1. Freeze the model and input contract

Before changing provider options, record the model file's hash, ONNX Runtime version, target OS, and device. Include input names, tensor types, shapes, and preprocessing rules. Keep a few representative input fixtures and expected outputs.

For an image model, “224 by 224” is not enough. Record channel order, color conversion, normalization, layout, and whether cropping occurs before resizing. A fast run on incorrectly normalized input is not useful performance evidence.

Use the same fixtures for the CPU baseline and accelerated candidate. Define acceptable numerical differences using the downstream task: classification agreement, a regression tolerance, or another justified criterion. Do not declare correctness solely because both configurations produce an output tensor of the expected size.

The ONNX Runtime mobile documentation covers mobile integration options. Pin the selected package and keep its headers aligned with the binary rather than mixing examples from a newer release into an older runtime.

2. Configure the provider from Swift

The Objective-C package exposes Swift interfaces. The example below uses the documented typed Core ML options and creates session options only; your existing inference owner must pass the result when it constructs its ORTSession.

import onnxruntime_objc

func makeSessionOptions(useCoreML: Bool) throws -> ORTSessionOptions {
    let options = try ORTSessionOptions()
    try options.setLogSeverityLevel(.verbose)

    if useCoreML {
        let coreML = ORTCoreMLExecutionProviderOptions()
        coreML.useCPUOnly = false
        coreML.useCPUAndGPU = false
        coreML.onlyEnableForDevicesWithANE = false
        coreML.onlyAllowStaticInputShapes = false
        coreML.enableOnSubgraphs = false
        coreML.createMLProgram = true
        try options.appendCoreMLExecutionProvider(with: coreML)
    }

    return options
}

This assumes a package exposing these options and an OS supporting ML Program. createMLProgram requires Core ML 5 or later. The typed interface also allows a controlled CPU-only Core ML experiment and a CPU/GPU configuration excluding ANE. See the options reference.

For comparison, construct a separate session using makeSessionOptions(useCoreML: false). That omits explicit Core ML registration; retain the standard CPU provider in the runtime build. A Core ML session configured to use CPU only is a different experiment from running ONNX Runtime's CPU provider.

The session-options API documents registration and the newer dictionary-based Core ML interface. Use the interface actually shipped in your pinned package. Do not translate Python option names into guessed Swift properties.

Verbose logging is for a controlled diagnostic build. Reduce logging for release, and inspect logs before sharing them because model paths and other implementation details may appear. Keep session construction away from latency-sensitive UI work and measure it separately from inference.

3. Check registration, partitioning, and hardware separately

The Core ML provider documentation lists build requirements, supported operations, and configuration options. A package without Core ML support cannot gain it through a runtime flag. An unsupported operation or input shape can prevent a graph region from being delegated.

Investigate in this order:

1. Registration: Did appending the provider succeed? Preserve initialization errors rather than silently replacing a failed accelerated session with a CPU session.
2. Model loading: Did the selected format and model load successfully? Record construction time and compilation-related failures separately.
3. Assignment: Which graph regions are delegated? Read the runtime's diagnostic output and investigate the specific unsupported operation, shape, or type.
4. Execution: How much time is spent in preprocessing, delegated work, remaining operators, and postprocessing?
5. Hardware: Use the available Core ML and system profiling tools on the device to investigate CPU, GPU, and Neural Engine behavior. Provider assignment alone cannot establish ANE execution.

Some CPU activity is expected in an app that prepares tensors, manages sessions, and renders output. A CPU graph in Instruments is not, on its own, evidence that acceleration failed.

4. Change one variable per experiment

For a model with truly fixed inputs, compare an export whose shapes express that fact. Enabling a static-shape-only option does not rewrite a dynamically shaped model into a static one. It changes eligibility for delegation. Keep this experiment separate from format, quantization, and compute-unit changes so the result has an identifiable cause.

Likewise, more delegated nodes are not automatically better. A small amount of useful accelerated computation surrounded by conversion overhead may not improve the user-visible operation. Optimize the complete feature, not a percentage in a log.

5. Capture an inspectable profile

ONNX Runtime supports JSON latency profiles with operator and threading information. Its profiling documentation describes the supported tooling and trace viewers.

For an initial model investigation on a Mac, the following Python diagnostic enables profiling and reads named input tensors from a prepared NumPy fixture. It is not an iPhone benchmark or a substitute for validating the iOS package.

import numpy as np
import onnxruntime as ort

if "CoreMLExecutionProvider" not in ort.get_available_providers():
    raise RuntimeError("This Python runtime has no Core ML provider")

options = ort.SessionOptions()
options.enable_profiling = True
session = ort.InferenceSession(
    "model.onnx",
    sess_options=options,
    providers=["CoreMLExecutionProvider", "CPUExecutionProvider"],
)

# Each NPZ key must match an ONNX input name. Prepare the fixture using
# the same dtype, shape, and preprocessing rules as the shipping app.
with np.load("inputs.npz", allow_pickle=False) as fixture:
    inputs = {item.name: fixture[item.name] for item in session.get_inputs()}

session.run(None, inputs)
print(session.end_profiling())

Use the Python runtime's InferenceSession API for session and input inspection. Confirm the provider used in the trace, not just its appearance in the available-provider list. Repeat on the target app using the diagnostics exposed by its selected binding and build.

6. Measure what the person waits for

Record these measurements for each device/configuration pair:

• Session construction time from a cold start.
• First inference after construction.
• Warm inference distribution over repeated representative inputs.
• Total time from the user's action to usable output, including preparation and decoding.
• Peak memory and behavior during a sustained series of requests.
• Output correctness against the chosen acceptance rule.

Report the number of samples and both a typical and tail latency, such as median and p95. Keep device thermal conditions, build mode, and model version in the report. Comparing a simulator Debug run with a physical-device Release run tells you little about provider choice.

Do not publish a speedup unless you measured it. A useful initial report can say that registration works, identify remaining CPU regions, and state that device performance is still unmeasured. Those are distinct milestones.

Ship the configuration that passes the feature's requirements

Before release, verify that the model and required external weight files are included, inference works without network access if promised, and initialization failure has a deliberate user-facing outcome. Confirm the oldest supported device meets the product's latency and memory budget.

Keep a CPU fallback only if its measured behavior is acceptable for the feature. If it takes too long, use a smaller workload or an explicit unavailable state instead of freezing the interface. Do not silently upload content to a cloud model as a substitute for local inference.

For broader product decisions, connect these measurements to the iOS app tech stack decision guide. A provider setting is successful when it improves a correct, maintainable customer workflow on the devices you support.