Most OTA disasters share a common root cause: the update mechanism was treated as a feature to ship once, rather than infrastructure that has to survive every possible failure — power loss mid-write, a corrupted download, a bad image that boots into a crash loop, or a device that drops connectivity halfway through. A design that only handles the happy path works fine in testing and fails expensively in the field, at scale, usually on the update that matters most.
Dual-bank (A/B) partitioning is the foundation
The single most important design decision is whether the device writes new firmware directly over the running image, or into a separate, inactive partition that only becomes active after it's verified. Direct-overwrite designs are simpler but have no safe fallback — if power is lost mid-write, the device has no valid firmware to boot into. A dual-bank (A/B) scheme writes the new image to the inactive bank, verifies it, and only then switches the boot pointer, leaving the previously-working bank intact as a fallback the entire time. The flash cost is real — roughly double the firmware storage — but it's what makes a failed update recoverable instead of fatal.
Verify before you trust
- Cryptographic signature verification on the received image before it's ever marked bootable, preventing both corrupted transfers and unauthorized firmware from being installed
- Checksum/hash validation of the complete downloaded image, not just individual chunks, since chunk-level checks can miss ordering or truncation errors
- Version and hardware-compatibility checks baked into the image metadata, so a firmware build meant for one hardware revision can't be installed on an incompatible one
The rollback plan matters more than the update plan
A new firmware image that passes signature checks can still be functionally broken — a bug that only manifests after boot. The critical design pattern here is a watchdog-backed boot confirmation: the device boots the new image provisionally, and only marks it permanently active after it confirms successful operation (a known set of self-checks passing, a successful check-in with the backend, or similar). If that confirmation never happens within a timeout, the bootloader automatically reverts to the last known-good bank on the next boot. Without this, a subtly broken update can brick an entire fleet with no remote recovery path.
Failure modes an OTA pipeline needs to survive
- Power loss at any point during download, write, or the boot-switch itself
- Connectivity dropping mid-download, requiring resumable transfers rather than restart-from-zero
- A signed, valid image that is nonetheless functionally broken
- Devices that are offline for extended periods and receive multiple version-skips' worth of updates at once
- Concurrent updates across a large fleet overwhelming backend bandwidth or triggering unintended simultaneous reboots
Staged rollouts catch what testing doesn't
Even a well-tested update can behave differently across the diversity of real-world field conditions than in a lab. Rolling an update to a small percentage of the fleet first, monitoring check-in health and error rates, and only expanding once that cohort is stable is standard practice for any fleet past a trivial size — and it turns a potential mass incident into a contained one.
How we approach this
OTA infrastructure gets designed alongside the firmware architecture from the start, not bolted on before launch — dual-bank partitioning, signature verification, and watchdog-backed rollback are treated as required, not optional. See our firmware work and cloud & device platform work for how update infrastructure fits into a full fleet-management build.