On-Device AI Systems: Quantization, Privacy, Updates, and Evaluation

A model that runs on a phone is not automatically a successful mobile feature. It may fit the application package yet exceed peak memory after camera buffers are allocated. It may have excellent benchmark latency while falling back to a slower CPU path on the devices customers actually own. It may keep raw inputs off a server but still send revealing logs through analytics. And a model that was accurate on a held-out dataset can degrade after conversion, an operating-system change, or a rushed update.

Planning on-device AI means treating the model, runtime, application, hardware, privacy boundary, and release mechanism as one deployed system. The practical objective is not to maximize a model metric in isolation. It is to meet a user-facing quality goal within explicit constraints for latency, memory, energy, availability, privacy, and maintainability. Official mobile and model-runtime guidance recognizes both the benefits and trade-offs: local inference can avoid network waits and continue offline, while consuming device resources and adding model-size and integration costs. 1 2

This article focuses on the deployment of inference on mobile and embedded devices. It does not treat every device-connected service as an edge-computing architecture problem. The central questions are more concrete: how much compression is acceptable, which operations execute where, what happens under thermal or memory pressure, what information crosses the device boundary, how can a release be reversed, and how do tests represent a diverse device fleet?

Begin with the user-visible contract

Before selecting an architecture, write down what the feature promises. “Works offline” is not equivalent to “works in every state without a network.” An app might install a small baseline model locally, then download an optional language pack; the core action is available offline, but a newly selected language is not until installation finishes. “Private” should identify which inputs are processed locally, which derived values may leave the device, and what happens during error reporting. “Fast” should specify a user-perceived path, such as time from shutter press to a usable result, rather than a model-only average.

Turn those claims into conditions that can be tested. For a camera classifier, that might mean a defined set of supported image dimensions, response time under a representative thermal state, a maximum acceptable memory footprint alongside the app, and a clear no-network fallback. For speech recognition, it may mean accuracy across relevant accents and background noise, plus a stated behavior when a language model is missing. The exact limits are product-specific. There is no universal latency, accuracy, or battery threshold that applies to every application, and benchmark results should not be presented as guarantees for devices outside their measured conditions.

The contract must also distinguish required from optional behavior. If local recognition is the primary experience, a server outage should not disable it. If a larger cloud model supplies an enhancement, the app should disclose that the enhancement requires a connection and decide whether the input is ever sent. If a user declines transmission, the feature should remain understandable rather than silently retrying through another route. These choices should be embodied in app states and tests, not left in a product slide or privacy statement alone.

Map the whole inference path before optimizing. A common vision path includes camera capture, image conversion, resizing, normalization, model execution, post-processing, and rendering. The model’s reported execution time measures only one segment. Copying an image into a different pixel layout, waiting for a GPU queue, or blocking the UI thread can dominate the user’s actual wait. Similarly, an audio pipeline may spend substantial energy capturing and buffering before inference begins. Instrument each boundary so the team knows which stage consumes time and memory.

The most useful initial model is a known-good floating-point version in the intended runtime, not an unconverted research checkpoint. LiteRT guidance recommends establishing a float-converted baseline to confirm operator compatibility and to isolate subsequent quantization errors. 3 That baseline can be slower or larger than the eventual production artifact; its purpose is diagnostic. If it already misbehaves, quantization is not the first problem to solve.

Compression is a family of compromises

Quantization approximates numerical values with lower-precision representations. It often reduces the storage needed for weights and can enable efficient integer kernels, but it does not produce a free, universal speedup. The result depends on the network, operators, tensor shapes, runtime, accelerator, and conversion settings. A compressed file is evidence of smaller storage, not proof of lower end-to-end latency, lower peak memory, or better energy efficiency.

Post-training dynamic-range quantization typically quantizes weights while preserving floating-point interfaces. LiteRT describes this route as requiring no representative calibration dataset; activations may be dynamically quantized for certain operations, while outputs remain floating point. Its documented optimization summary associates it with smaller model size and CPU acceleration, but those figures describe the tool’s stated outcomes, not a promise for each target device or model. 3 This option is often a sensible early experiment because it minimizes calibration work and can leave application interfaces unchanged. It may be inadequate when an integer-only accelerator requires a fully quantized graph.

Full integer quantization also represents activations using integer values. Since activation ranges depend on inputs, the converter needs representative examples to estimate ranges. LiteRT describes a calibration subset on the order of hundreds of samples as a possible starting point, not as a sufficient recipe for every task. 3 The examples must cover the application’s real input distribution: lighting, camera processing, languages, audio levels, image sizes, and the kinds of unusual but valid cases users produce. A calibration set drawn only from clean training examples can produce ranges that clip deployment inputs or waste precision on irrelevant extremes.

Calibration is a data-selection decision as much as a converter setting. Keep a provenance record for the calibration sample set, its preprocessing, label or subgroup coverage, and the precise conversion configuration. Use data that the team is authorized to use. In sensitive domains, calibration can often be performed within the controlled training environment; it need not involve collecting additional raw data from deployed phones. The representative set influences the numerical mapping, so changing preprocessing or input normalization after calibration can invalidate assumptions even if the graph still loads.

Quantization assigns a scale and zero point that map real values to integer values. Rounding and clipping introduce approximation. Errors can accumulate through layers, particularly when a model is small, activations have outliers, or a task depends on subtle values. A model can retain high overall accuracy while losing performance for a specific class, language, lighting condition, or user subgroup. Therefore compare the converted artifact against the float baseline on a fixed evaluation set, and inspect more than one aggregate metric. For classification, review per-class recall and confusion; for detection, examine localization quality and missed-object rates; for speech, evaluate relevant word error patterns and noise conditions.

Float16 weight quantization is another path. It can reduce the weight storage requirement while keeping a floating-point computation style. LiteRT notes that a GPU delegate may operate on float16 data directly, whereas CPU execution can dequantize weights to float32. 3 Thus, a float16 model that performs well on a compatible GPU may not be the best CPU artifact. The same file may exhibit different time, memory, and power characteristics according to which backend actually executes it.

Quantization-aware training (QAT) simulates low-precision effects during training, allowing the model to adapt to them. It is a response when post-training conversion causes unacceptable quality loss, not an automatic upgrade. QAT entails changes to the training workflow, retraining, validation, and reproducibility controls. LiteRT documentation recommends considering it when post-training quantization accuracy loss is too high. 3 Compare the development cost with alternatives such as selecting a more robust architecture, retaining higher precision in sensitive operations, or accepting a modestly larger model. Mixed precision and selective quantization may preserve quality, but require verifying operator support and actual delegate behavior.

Compression can also involve pruning, weight sharing, architecture changes, and distillation. These techniques have different effects. Removing weights does not guarantee a smaller or faster executable if the runtime stores a dense tensor and still executes dense kernels. A structured sparsity pattern can be more useful when hardware and kernels explicitly exploit it. Distillation may yield a smaller student, but quality then depends on training data and teacher behavior. An architecture designed for the intended runtime may outperform a larger network compressed after the fact. Each candidate needs the same system-level measurement protocol.

The right optimization order begins with a clear baseline and then isolates one change at a time. Convert without quantization; validate outputs. Apply one quantization mode; validate again. Inspect model size, operator types, conversion warnings, and runtime partitioning. Run the same evaluation set and device suite. If several transformations are applied together, a regression becomes difficult to attribute. Keep model artifacts versioned with their converter, source checkpoint, calibration data reference, and evaluation report so a performance or quality change can be traced to its cause.

Runtime compatibility is part of model design

A model file is not an executable promise that every operation will run efficiently on every processor. A runtime parses the graph and dispatches operations to available kernels or accelerators. Some operations may be unsupported by a delegate, require a particular data type, or use a shape that is inefficient for that backend. An accelerator might execute a supported prefix while unsupported operations fall back to the CPU. Crossing those boundaries can add copies, synchronization, and conversion overhead. A nominally accelerated graph may therefore be slower than a consistent CPU path.

Treat the target runtime as a constraint during model selection and conversion. Record supported operator versions, tensor layouts, dynamic-shape behavior, precision modes, and delegate requirements. Run model validation on actual production runtimes, not just in a desktop converter or simulator. Check logs for delegation and fallback, and make that information available in test reports. An integration test should fail when a critical operation unexpectedly changes execution provider or when a model no longer loads after a dependency update.

Android’s NNAPI documentation illustrates why assumptions about hardware access age quickly: the API is deprecated, and Android recommends alternative solutions for performance-critical workloads. It also describes how runtime behavior can depend on drivers, available processors, and supported memory-buffer configurations, and instructs developers to test target devices. 2 The deployment lesson is not that one specific API should always be used. It is to avoid making an application’s performance contract depend on a single vendor path whose availability or lifecycle is outside the app team’s control.

Build a small compatibility matrix around classes of hardware and software that matter: operating-system versions, processor families, memory tiers, runtime versions, and relevant accelerator backends. Individual product models within a family can still differ in drivers and firmware, so categories are a sampling strategy, not a claim of uniformity. Include a CPU fallback path when practical. It may be slower, but it can preserve correctness and availability on unsupported devices. Decide whether that fallback meets the feature contract or whether the app should defer the operation, disable a capability, or offer a cloud alternative with consent.

Input and output conventions are common sources of apparently mysterious model failures. A model can expect RGB while the camera API supplies a different channel order; it may require normalized floats while application code sends byte values; output indices may need a label map that has changed independently. Define the interface as a versioned contract: tensor names, shape, element type, color space, normalization, padding, tokenization, label order, and output interpretation. Validate it at the application boundary. Avoid relying on undocumented assumptions in the training notebook.

Pre- and post-processing belong in the same compatibility review as neural-network operators. A converter can preserve a neural graph yet the end-to-end app can break because an image resize changed interpolation, a tokenizer was updated, or a post-processing threshold shifted. If the model and a vocabulary, normalization table, or label map must move together, package them as one logical release. Apple describes model collections delivered atomically for related Core ML models; the same consistency principle applies to a model plus its app-side metadata. 4

Fit the work inside memory, thermal, and battery limits

Model size on disk is only one resource measure. At execution time, memory can include model weights, runtime allocations, intermediate activations, input and output tensors, camera or audio buffers, compiled kernels, and app state. Peak use may occur during initialization or while two versions coexist during an update. On devices with limited memory, the operating system may reclaim resources or terminate the process. Measure cold start, warm inference, concurrent application state, and update installation rather than estimating resident memory from the compressed file alone.

A useful memory budget separates persistent and transient components. Persistent costs include the loaded model and runtime state. Transient costs include activation buffers and preprocessing copies. Reuse buffers when safe, avoid retaining full-resolution inputs longer than necessary, and stream work in bounded chunks where the model permits it. But buffer reuse can introduce concurrency bugs if one invocation overwrites data that another still needs. Establish ownership and lifetime rules, then test concurrent requests and cancellation explicitly.

The peak is often shaped by data movement. Resizing a large image can allocate both source and destination arrays; converting to a model layout can create another copy. A theoretically compact quantized model can still incur a large float conversion buffer if an operation is not supported in the chosen precision. Measure memory around the whole pipeline with representative input sizes. Test the largest valid input, not merely the common case, and place sensible bounds on inputs that otherwise allow unbounded allocation.

Latency must be measured as a distribution. Report cold model load separately from steady-state inference, and distinguish median from tail behavior. A camera experience may need predictable frame-to-frame work, while an occasional document scan can tolerate a longer first-run compilation if it is disclosed or performed off the interactive path. Record preprocessing and post-processing time, queue wait, and UI presentation. The device may be doing other work, so isolated laboratory measurements should be complemented by realistic app-load tests.

Thermal and energy behavior differ from a brief benchmark. Sustained inference can heat a device and lead to throttling, making later predictions slower. High-frequency camera inference can also compete with display, networking, and other app tasks. Android’s guidance explicitly calls out battery use from neural-network evaluation as a consideration. 2 A production test should include repeated inference under a sustained workload, observe timing over time, and record device temperature or thermal state where the platform exposes it. Do not extrapolate the first run to a long session.

Scheduling is a product decision. A model need not run on every frame if a lower cadence, change detector, or user-triggered action meets the requirement. Batching may improve throughput but increase memory and delay the first result. Lowering resolution may reduce computation but make small objects undetectable. These changes should be evaluated against task quality and response needs. For periodic sensing, duty cycling or early exit can conserve energy; for accessibility functions, reducing work must not make the feature unreliable when it matters.

Account for charging, battery saver, background execution limits, and interruptions. If a lengthy inference is not urgent, defer it when resources are constrained. If it is urgent, make the user impact explicit and keep the workload bounded. A robust app handles the operating system suspending or killing work, avoids corrupt partial results, and can resume safely. Cancellation is not a rare exceptional path on mobile; it is part of normal lifecycle behavior.

Offline capability needs an explicit boundary

Local inference eliminates the need for a network round trip during the inference step, but the feature may still depend on connectivity for model downloads, licenses, account access, remote metadata, or server-side enrichment. Document which operations are available immediately after installation, which require a download, and what happens when connectivity disappears midway. An app should not label a feature offline if its first-use setup silently depends on a server.

Decide whether the model is bundled or installed separately. Bundling gives a known baseline that ships with the application and can be used immediately, but it increases package size and ties model changes to app releases. Downloading after installation keeps the initial app smaller and allows independent model releases, but introduces download failure, storage pressure, integrity checks, and compatibility coordination. Apple’s Core ML documentation describes downloading and compiling a model on the device, while its deployment session outlines independent model delivery and targeting. 4 5 The choice depends on user expectations and model footprint, not on a universal preference.

A practical hybrid is to include a small, known-compatible baseline and offer optional assets for capabilities that justify their size. The app should make download state visible, respect metered connections and user settings, verify the artifact before activation, and retain enough safe fallback to continue core functionality. Disk quotas and cleanup policy matter: model packages can accumulate across versions, languages, or user profiles. Never delete the only working artifact before confirming that the replacement is complete and loadable.

Offline behavior must include failure modes, not just a success demonstration in airplane mode. Test first launch without a connection, loss of connectivity during a download, interrupted installation, low storage, clock or authentication edge cases if relevant, and recovery after reconnecting. Distinguish “model unavailable” from “prediction failed” in state handling. If the UI retries automatically, use bounded retries and avoid repeated expensive initialization that drains battery.

Privacy is a system boundary, not a location label

Running inference locally can keep raw input on the device, which is a meaningful privacy advantage. Apple describes Core ML as using app data for prediction on the person’s device, and Android describes on-device inference as avoiding transmission of data away from the device. 1 2 However, local computation by itself does not establish that a feature is private. Inputs can be copied to application logs, crash reports, analytics events, temporary files, backups, or a cloud fallback. Outputs can reveal sensitive attributes even when the original image or text is never uploaded.

Draw a data-flow diagram for the feature. Include camera or microphone APIs, preprocessing buffers, model inputs and outputs, persistent caches, crash diagnostics, analytics, application servers, third-party SDKs, and support workflows. At every crossing ask what data moves, why it is needed, how long it remains, who can access it, and how a user can control it. A diagram that stops at the model invocation misses the places where sensitive content often escapes the intended boundary.

Separate operational telemetry from content telemetry. It may be useful to know that model version 12 failed to load on a certain runtime family or that inference exceeded a latency band. That does not automatically justify collecting a user’s image, transcript, prompt, or full prediction vector. Prefer aggregated counts, coarse performance buckets, and error codes that do not expose content. Scrub exception messages and avoid attaching raw input to crash reports. If content collection is necessary for a specific quality program, define a separate purpose, explicit permission where required, retention and access controls, and a way to decline without losing the core local feature.

Derived data deserves attention. Embeddings, classifications, confidence scores, recognized text, and personalized model parameters can remain sensitive even when they are not human-readable in the same way as the source input. A telemetry event with a rare label sequence or a stable identifier can reveal user behavior. Minimize identifiers, aggregate where possible, set retention periods, and test actual payloads in a network proxy during development. A privacy review should inspect traffic generated by the full app, including third-party libraries and background retries.

Local storage creates its own risks. Temporary inputs should be deleted when no longer needed; caches should have defined lifetimes; sensitive model personalization data should be protected according to its sensitivity. Model confidentiality and user-data confidentiality are distinct concerns. Apple documents model encryption capabilities for protecting a model artifact, but model encryption does not encrypt every application input, and it does not prevent a determined party from observing runtime behavior or outputs. 5 Decide whether protecting intellectual property, protecting user data at rest, or both are requirements, and apply controls to the corresponding assets.

A cloud fallback changes the privacy boundary even if it happens rarely. Define the precise conditions for fallback, the data fields transmitted, the service endpoint, and the user notice or consent behavior. Do not silently route a request to a server because the accelerator is absent or a local model fails. Consider a local reduced-quality mode, queueing the task until a supported model is available, or asking the user before transmission. Ensure privacy settings apply consistently to diagnostic and fallback paths.

On-device training or personalization can reduce the need to centralize individual data, but it introduces risks around model state, reset behavior, poisoning, and accidental cross-account reuse. Treat personalized weights as user data. Scope them to the correct profile, provide reset or deletion behavior, test account switching, and guard against copying stale state into shared storage. Any synchronization of personalized parameters is a separate data flow that requires its own threat and privacy analysis.

Model releases need software-grade controls

A model update is executable behavior, not merely a content refresh. It can change classifications, decisions, safety behavior, memory use, or the information a system emits. Give each artifact an immutable version and record its source checkpoint, conversion settings, runtime compatibility, evaluation results, and intended application versions. Link it to a release decision and an owner. If the model changes without a corresponding metadata change, the artifact should still be identifiable after it reaches a device.

For separately delivered artifacts, validate authenticity and integrity before activation. A transport channel protected by TLS is not a complete artifact-verification design: the app should verify a signature or another trusted cryptographic digest anchored in an application-controlled trust mechanism. Protect signing keys, restrict release authority, and preserve audit records. The verification procedure should cover the model and any coupled label maps, vocabularies, configuration, or code-like assets that influence interpretation.

Install updates transactionally. Download into a temporary location, check available storage, verify the artifact, validate its manifest and compatibility, and attempt to load it before switching the active pointer. Keep the currently working model until the candidate passes those checks. An interrupted download should leave the old model usable. If the operating system or runtime rejects the new model at load time, the app should report a controlled failure and continue with the prior artifact or a baseline.

Rollback is a first-class path. Maintain an explicit known-good version and define who can halt a rollout. Rollback may mean switching to a previous signed model, disabling a problematic model-driven feature, or shipping an app hotfix if the interface contract itself changed. The app needs to know which versions remain compatible with its current code and data. A model that loads but yields invalid outputs should be treated as unhealthy, not as a successful update. 8

Apple’s Core ML materials describe model deployment independent from the app update cycle, including targeting device populations, and atomic delivery of related model collections. 4 Independent release is valuable because a model can be corrected without waiting for an application build, but it also increases the need for compatibility controls, staged rollout, and rollback. Bundled models have fewer independent moving parts, though an urgent model correction then depends on the application release process.

Use staged deployment. Begin with internal devices and a small representative cohort, then expand only after checking load failures, latency, resource behavior, and task quality signals. Cohorts should cover the hardware and software combinations that matter, not just a random slice that happens to overrepresent new devices. Keep the prior artifact available during the rollout. A release dashboard should distinguish download success from activation success and actual inference health.

Avoid assuming that online telemetry can safely establish model quality. Labels may not be available, feedback can be biased, and a model can fail silently while response latency appears normal. Define post-release signals before rollout: crash and load rates, fallback frequency, user corrections where appropriately collected, opt-in quality feedback, and task-specific tests run on representative devices. Do not treat a rising usage metric as proof that predictions are correct.

Evaluate the deployed system, not only the checkpoint

Evaluation begins with a fixed, versioned test suite that reflects the intended use. It should include the ordinary inputs, boundary cases, degraded sensor conditions, and important subgroups. Keep training, calibration, and evaluation roles distinct to avoid tuning conversion settings against the only test set and then reporting its result as independent evidence. Record preprocessing and label definitions alongside data version. A metric without its data and evaluation code is difficult to reproduce.

Measure quality before and after conversion. Compare floating-point model, converted model in a reference runtime, and the artifact running through the application’s production path. Differences between the second and third stages expose integration bugs such as normalization mismatches or output mapping errors. For quantization, inspect task-level outcomes plus numerical differences in selected intermediate tensors when practical. Large tensor-level differences do not always imply a user-visible problem, while a small shift around a decision threshold may cause one.

Use metrics that match the product harm. Accuracy can conceal rare but costly misses in imbalanced tasks. A false negative in a safety alert is not interchangeable with a false positive in an optional photo tag. Set acceptance criteria with product and domain owners, assess subgroup performance, and examine confidence calibration if the interface uses confidence to decide when to abstain or ask for confirmation. Avoid inventing a universal acceptable quantization drop; it depends on the use case, error costs, and relevant population.

For systems comparisons, MLPerf Tiny provides a useful model of multidimensional evaluation. Its benchmark suite reports task quality and single-stream inference latency, with optional energy measurements, and defines workloads with dataset and quality targets. 6 Its paper frames accuracy, latency, and energy as linked trade-offs for tiny systems. 7 These standardized results help compare systems under specified conditions, but they are not substitutes for evaluating the application’s data, user experience, sustained workload, privacy requirements, or device distribution.

Benchmark reproducibly. Record device model, operating-system build, runtime and delegate versions, power mode, thermal state, input dimensions, warm-up policy, number of runs, and whether preprocessing is included. Report distributions and distinguish cold from warm behavior. State whether the tested build uses production compiler flags and the same model artifact as release. If measurements are collected while a device is charging or in a laboratory cooling setup, say so; those conditions may not represent ordinary use.

Measure energy with a method suitable to the device and workload. A short inference can have small signal relative to background consumption; repeated trials and a consistent measurement setup improve interpretation. Track energy per task or per sustained session rather than relying solely on CPU utilization. The MLPerf Tiny paper explicitly includes energy because speed and model accuracy alone do not describe the resource trade-off. 7 Where an external power instrument is unavailable, platform counters can still be useful for relative comparisons, but document the method and do not imply that it is a direct measurement of battery capacity consumed by a user session.

Device cohort QA should be designed around failure diversity. Select representative devices by meaningful differences in system memory, processor capability, accelerator and driver, OS level, and deployment prevalence. Include older supported devices as well as current high-end devices. A “representative” cohort is not just a list of popular phone brands: it should cover distinct runtime paths and resource limits. Test multiple physical devices where vendor variation or firmware differences could matter.

Build a test ladder. On every model build, run conversion checks, interface-schema validation, and a small quality suite. In continuous integration, exercise runtimes and simulated constraints that are reliable in automation. On physical devices, test load, operator delegation, memory, latency, thermal behavior, and battery-relevant sessions. Before release, run the complete cohort suite and a staged rollout. After release, monitor operational health and keep a rapid disable or rollback route.

Include adversarial and malformed inputs. A corrupt image, empty audio buffer, unexpectedly long text, NaN tensor, unsupported shape, or invalid model output should not crash the app or leak content into logs. Bound input sizes, validate outputs before using them, and provide a safe response for low confidence or failed inference. For generative models, evaluate prompt-injection and unsafe-output handling at the product layer as well as model quality; keeping generation local does not remove the need for appropriate safeguards.

Test lifecycle transitions. Switch apps during inference, lock the screen, rotate the device, revoke a permission, run low on storage, enter battery saver, interrupt a model download, and kill the process. Verify that the feature recovers and does not leave temporary sensitive files behind. For a model update, test both successful activation and failures at every stage. The most damaging defects often occur in the transitions between otherwise healthy components.

Choose local, cloud, or a deliberate hybrid

Local inference offers predictable availability when a network is absent, avoids network round-trip latency, and can reduce the need to transmit sensitive raw inputs. Its costs include constrained compute and memory, battery use, model distribution, device fragmentation, and the need to support multiple runtime paths. Official Android and Apple descriptions emphasize the local latency, availability, privacy, and power trade-offs rather than presenting local execution as universally superior. 1 2

Cloud inference can provide access to larger models or centralized updates, and it simplifies consistency when a service needs a single controlled runtime. It also depends on connectivity and service availability, adds network delay, incurs server and transfer costs, and moves data across a trust boundary. These are architectural trade-offs; they do not imply that cloud models are always more capable or that on-device models are always more private. Actual quality and privacy depend on the specific system and data practices.

A hybrid design should specify routing policy in plain terms. For example, a small local model might handle routine classification and reject uncertain inputs; an optional server request might handle a complex case only after the user agrees. Another design might perform local filtering and send a minimized representation. Such a representation may still disclose sensitive information, so it should not be assumed anonymous. Define what happens when the network fails, the cloud response is delayed, the local model is unavailable, or the two disagree.

Use a decision matrix built from measured requirements: minimum offline capability, acceptable quality, latency distribution, peak memory, sustained energy, update frequency, regulatory or contractual constraints, and total cost. Prototype at least one local path on target devices before committing to a cloud-only dependency or an accelerator assumption. Prototype the network path under poor connectivity before committing to a cloud fallback. A hybrid can increase complexity because teams must test more states and keep policies consistent; it is not automatically the best of both worlds.

A practical deployment review

Before releasing an on-device model, confirm that the model is useful in the full application path, not just accurate as a checkpoint. The team should be able to answer: what is the baseline artifact; what transformations were applied; how representative calibration was assembled; which runtime and operators are required; what executes on CPU, GPU, or a neural accelerator; and how fallback behaves when a delegate is unavailable?

Confirm that the resource budget reflects peak memory during real app use and repeated inference, not only file size or one warm run. Confirm that users can understand offline prerequisites and that optional model downloads fail safely. Check what data leaves the device during success, failure, analytics, crash reporting, and cloud fallback. A local inference path with content-rich diagnostics is not a local-only data design.

For release readiness, verify artifact identity, integrity checks, compatibility metadata, staged rollout, rollback, and the ability to disable a malfunctioning model. For quality readiness, compare float and converted versions on a fixed suite, measure important subgroups, test the production preprocessing path, and execute on device cohorts that represent distinct hardware and runtime behavior. For operations readiness, ensure that health signals do not overcollect sensitive content and that the team can detect a silent loss of acceleration or a spike in resource use.

No single compressed model, accelerator API, privacy label, or benchmark result settles these questions. Good deployment is a continuing discipline: build a measured baseline, optimize one variable at a time, test real devices and real failure states, and preserve the ability to change course. Quantization is valuable when its quality and runtime effects are demonstrated. Local execution is valuable when its privacy and availability boundaries are real. Updates are valuable when they are authenticated, compatible, observable, and reversible. Evaluation is valuable when it resembles the conditions under which people will actually use the feature.

References

[1] Android Neural Networks API [2] Core ML Documentation [3] LiteRT Post-training Quantization [4] Use Model Deployment and Security with Core ML [5] ML Kit Guides [6] MLPerf Inference Tiny Benchmark [7] MLPerf Tiny Benchmark Paper [8] NIST SP 800-124 Revision 2: Guidelines for Managing the Security of Mobile Devices in the Enterprise