A model trained once, evaluated well, and deployed to a fleet is the easy part. The harder, ongoing problem is keeping it accurate as field conditions change — new failure modes appear, sensor behavior drifts, usage patterns shift — while ensuring a bad model update never silently degrades every device's behavior at once. That's what fleet-wide MLOps actually has to solve, and it borrows structure from firmware OTA practices as much as from traditional ML tooling.
Why fleet-deployed models need retraining, not just monitoring
A model's accuracy is only as good as its training data's resemblance to what it sees in production. For fleet-deployed models, that resemblance erodes over time — sensor hardware revisions change signal characteristics, new device configurations appear in the field, usage patterns evolve, and edge cases invisible in the original training set start showing up at scale simply because the fleet has grown. Left static, a model's real-world accuracy degrades gradually and often invisibly, until it's noticeably wrong. Planning for retraining as a recurring process, not a one-time step, is the starting assumption.
Central training, distributed inference
The standard pattern separates where a model is trained from where it runs: training happens centrally, using aggregated (and appropriately anonymized/derived) data from the fleet, with the compute resources a full training run needs; inference happens on individual devices, at the edge or in a lightweight cloud service, using whatever compute budget that layer has. This split means a model can be retrained on the full fleet's accumulated data without requiring every device to participate in training itself.
Versioning models like you version firmware
A model deployed to a fleet needs the same rigor around versioning that firmware does — every deployed model version tracked against which devices are running it, what training data and hyperparameters produced it, and what its evaluation metrics were before release. Without this, diagnosing a fleet-wide accuracy regression after the fact becomes close to impossible: there's no way to correlate "devices behaving oddly" with "which model version they're running" if that mapping wasn't tracked from the start.
Staged rollout — the same discipline as firmware OTA, applied to models
- Offline evaluation against a held-out test set before any deployment, using metrics that reflect the actual product decision the model informs, not just generic accuracy
- Shadow deployment — running the new model alongside the current production model on a subset of devices, comparing outputs without acting on the new model's decisions yet, to catch discrepancies before they affect behavior
- Staged rollout to a small device cohort, monitoring real-world performance signals, before expanding fleet-wide — mirroring the same staged approach used for firmware updates
- A rollback path that's as fast and reliable as a firmware rollback, since a model regression discovered after full rollout needs to be reversible quickly
What to monitor after a model is live
Accuracy metrics computed offline during training don't guarantee the same performance in production, where the model sees a different, evolving data distribution. Ongoing monitoring needs to track prediction-confidence distributions, rates of specific error types the product cares about, and — where feasible — proxy signals that correlate with real-world correctness even without ground-truth labels arriving in real time. A model that looks fine in the training report but drifts silently in the field is a common and hard-to-catch failure mode without this kind of ongoing observability.
How we approach this
We treat model deployment infrastructure with the same rigor as firmware OTA — versioned, staged, monitored, and reversible — rather than a one-off training exercise. See our AI & Intelligence work and cloud & device platform work for how this fits into a full fleet-management build.