A decision-first model that works cleanly on a laptop or in a browser tab does not automatically behave the same way on an Android device. Memory budgets are tighter. Thermal limits appear under sustained use. Battery impact becomes visible. Runtime choices (ONNX Runtime Mobile, TensorFlow Lite, MediaPipe, or vendor-specific accelerators) introduce new numerical and performance differences. Background restrictions and OS-level memory management can pause or kill the process that holds the model.
The decision itself—input in, label and confidence out—remains the same. Everything around that decision changes enough that measurements taken on desktop or in a desktop browser cannot be reused without re-validation. This post describes the practical differences observed when moving a decision model onto Android and the measurement points used on The Decision Bench. It does not report specific latency or memory numbers from any single phone. Those figures are device-dependent and must be collected under stated conditions.
Runtime Options and Their Trade-offs
Several runtimes can execute a decision model on Android.
ONNX Runtime Mobile provides a direct path for models already exported to ONNX. TensorFlow Lite is common when the source model lives in the TensorFlow ecosystem. Some devices expose NNAPI or vendor delegates that can accelerate execution on the DSP, GPU, or NPU. Each choice affects binary size, supported operators, quantization paths, and numerical behavior.
I treat the runtime and the delegate as part of the experimental condition. A latency number measured with the CPU baseline does not transfer to an NNAPI delegate run. Confidence scores can also shift slightly when the execution path changes, so golden-input comparisons are repeated after any runtime or delegate switch.
Model format decisions made earlier (ONNX export, quantization, fixed versus dynamic shapes) reappear here with stricter constraints. Operators that were acceptable on desktop may be unsupported or fall back to slow paths on mobile. The fallback is often silent unless explicitly logged.
Memory Limits and Process Lifetime
Android applies more aggressive memory management than a typical desktop process. The system can reclaim memory from background apps, and low-memory conditions can kill the process that holds the model weights. Loading a large decision model into an activity or a long-lived service therefore requires explicit attention to peak memory and to the ability to release resources when the UI is no longer visible.
I record memory snapshots at consistent points: before model load, after load, after a series of inferences, and after explicit release. Device-level tools and Android’s own memory APIs provide the figures. The absolute numbers matter less than the incremental cost of the model and the stability of the process under repeated use.
On lower-memory devices the same model that ran comfortably on a flagship phone may force the app into frequent garbage collection or into process death. Those outcomes only become visible when the measurement is performed on the actual device class that the product intends to support.
Latency Under Real Device Conditions
Latency on Android is influenced by factors that desktop measurements often miss.
Thermal throttling can increase inference time after a sequence of rapid calls. Power-saving modes and battery level can change CPU or accelerator availability. The first inference after load may include additional compilation or caching work that later inferences avoid. Concurrent system activity (other apps, network, location) introduces further variability.
I therefore measure both single-inference latency and sustained sequences. Individual timings are kept so that tails remain visible. Cold-start (process or model just loaded) is separated from warm inference. The device model, Android version, battery state, and whether the app was in the foreground are written into the same log entry as the timing numbers.
Because decision-first models produce a compact output, the inference cost itself is usually stable once the model is warm. The surrounding device state is the larger source of variation. Logging that state is what makes later comparisons possible.
Thermal and Battery Effects
Sustained inference produces heat. On mobile devices that heat triggers frequency scaling and, in extreme cases, visible thermal warnings. Battery drain also becomes a product concern when the model is used repeatedly in a session.
I do not attempt precise joule-level accounting in early tests. Instead I record qualitative and simple quantitative signals: whether throttling was observed, whether surface temperature became noticeable, and rough battery percentage change over a controlled sequence of inferences. These notes are enough to flag problems that require deeper instrumentation later.
A model that is accurate and well calibrated but that forces the device into thermal throttling under normal use may still be unsuitable for the intended interaction pattern. The Decision Bench treats thermal and battery behavior as first-class deployment constraints rather than as afterthoughts.
Logging on Device
The same decision-logging discipline used on desktop and in the browser is applied on Android, with a few mobile-specific fields:
Device model and Android version
Runtime and delegate in use
Battery level and charging state at the start of the sequence
Foreground versus background status
Exact input
Primary label and confidence
Secondary scores when available
Per-inference wall-clock latency
Any runtime warnings, fallbacks, or errors
Logs are written to a local structured file or to a debug buffer that can be extracted after the session. Relying solely on on-screen display loses the record as soon as the app is closed. Keeping the per-decision record makes it possible to isolate the exact inputs that produced surprising scores or latency spikes once the model is running on device.
What Changes in Practice
Several differences appear consistently when a decision model moves to Android.
Preprocessing must be re-validated. Tokenization or feature code that lived in Python may have been reimplemented in Kotlin or Java; small discrepancies change scores.
Quantization is more common because of size and speed pressure. Calibration and accuracy must be re-checked after quantization.
Network is no longer required for inference, which is an advantage, but model delivery and updates become an application responsibility.
User-visible failure modes shift. Instead of a server timeout the user may see a slow first load, a sudden process restart, or a decision that simply never appears because the model was evicted from memory.
None of these issues invalidates the decision-first approach. They only require that the same measurement standards—visible input, visible decision, visible score, visible timing and conditions—are applied on the actual target devices.
What On-Device Success Does Not Guarantee
Successfully obtaining the expected label and score on a single Android device does not guarantee acceptable behavior across the device range the product must support. Lower-memory phones, older Android versions, and different accelerator availability all need their own measurement passes.
Calibration and accuracy figures from desktop or server runs do not automatically transfer. Numerical differences introduced by mobile runtimes and quantization can move both the top label and the confidence distribution.
Finally, on-device execution does not remove the need for graceful degradation. Load failures, out-of-memory conditions, and thermal throttling must be detected and handled so that the product can fall back to a safer path rather than silently producing no decision or a stale one.
How This Fits the Rest of the Site
Deploy Notes posts document the concrete environments a decision model can occupy. Bench Tests supply the methods for latency, memory, and calibration that are reapplied on Android hardware. Field Tests check whether the same decisions remain useful when the model runs entirely on the device inside a real mobile workflow. Failure Log entries capture the cases where mobile constraints produced high-confidence errors, unexpected latency, or missing decisions.
The governing rule does not change. Show the input. Show the decision. Show the score. Show the timing and the device conditions. When those elements stay visible, running a decision model on Android becomes a measured deployment choice rather than an untested assumption.
No comments yet — be the first to share a thought.