The question of where a model should run gets treated as an ideological choice more often than it should — teams default to cloud because it's familiar, or default to edge because it sounds more impressive, without actually working through the constraints that determine which one the product needs. Four factors usually settle it: latency tolerance, connectivity reliability, data sensitivity, and the cost structure at the scale the product expects to run at.
Latency: how fast does the decision need to be?
A round trip to the cloud — upload, inference, download — typically adds tens to hundreds of milliseconds depending on network conditions, even under good connectivity. For a feature like a voice assistant's wake-word detection or a safety-critical anomaly alert, that latency, and its variability, can be disqualifying. On-device inference removes the network hop entirely, which is why it dominates any use case where the response has to feel instantaneous or where a delayed decision has real consequences.
Connectivity: can the product assume a network is always there?
Cloud inference has a hard dependency the product inherits: if the device is offline, the feature stops working, unless there's a fallback. For products deployed in environments with unreliable connectivity — industrial sites, rural agriculture, moving vehicles — that dependency is a real liability. On-device inference keeps working regardless of network state, which matters far more for some product categories than others.
Data sensitivity and privacy
Sending raw sensor data (video, audio, biometric signals) to the cloud for every inference means that data leaves the device, which has real privacy, regulatory, and bandwidth implications. Running inference on-device and only transmitting derived results — an event flag, a classification, an aggregated metric — keeps the raw data local by architecture, not just by policy. For products handling anything privacy-sensitive, that architectural difference is often the deciding factor independent of latency or connectivity concerns.
Cost, at the scale that actually matters
Cloud inference cost scales with usage — every inference call has a marginal cost, which is manageable at low volume and can become substantial at fleet scale with frequent inference. On-device inference shifts that cost to hardware — a more capable MCU/SoC with enough compute for the model, paid once per unit — which can be cheaper at scale but requires the hardware decision to be made early, since it's not something you can retrofit into an already-shipped device.
Where cloud still wins clearly
- Fleet-wide pattern detection that needs data from many devices simultaneously — a single device can't see the pattern in isolation
- Large models that genuinely don't fit the memory/compute budget of realistic edge hardware
- Models that need frequent retraining based on aggregated fleet data, where cloud deployment means every device benefits from an update instantly
- Workloads without a hard latency requirement, where the cost and flexibility of cloud infrastructure outweighs the benefit of moving inference to the edge
The common real answer: both, split deliberately
| Layer | Typical placement |
|---|---|
| Time-critical local decisions | On-device |
| Fleet-wide trend/anomaly detection | Cloud |
| Privacy-sensitive raw data processing | On-device |
| Model training and retraining | Cloud |
A well-architected product often runs lightweight, latency-critical inference on-device while sending derived, non-sensitive signals to the cloud for aggregation, trend analysis, and periodic model retraining that then gets pushed back to the fleet via OTA. The split is a design decision made per capability, not a single answer for the whole product.
How we approach this
We work through this tradeoff per feature, not per product — deciding placement by the actual latency, connectivity, and privacy constraints in play, rather than defaulting to whichever is more familiar. See our AI & Intelligence work for how this fits into a full engagement, from model integration through fleet-wide deployment.