Edge AI vs Cloud AI: Where Should Inference Run?

Most smart devices need both edge AI and cloud AI

Edge AI runs a trained model on the device itself, or on a gateway next to it, so the device can make decisions without sending raw data to a server. Cloud AI sends the data to a data centre and returns the result. For most connected products the right answer is not one or the other: the device handles the fast, frequent, private decisions locally, and the cloud handles training, fleet-wide analysis and anything that needs a large model.

The real engineering question is where each piece of inference should run: on the microcontroller, on an application processor with an NPU, on a gateway or edge server, or in the cloud. This guide is written for teams that build hardware. It explains the difference between edge and cloud AI, the four places a model can run, how to decide, what TinyML makes possible on small chips, and how to keep models secure and up to date once devices are in the field.

What is edge AI?

Edge AI is the practice of running machine-learning inference on or near the device that collects the data, instead of in a remote data centre. The model is usually trained in the cloud, then optimised and deployed to the device, where it turns sensor data (audio, vibration, images, current, motion) into a decision or a small summary.

Typical examples:

A vibration sensor on a motor that detects an abnormal bearing pattern and sends only an alert.
A smart doorbell that recognises a person locally and uploads a short clip instead of a continuous stream.
A wearable that classifies activity or detects a fall on the device.
A smart meter that spots tampering or an unusual load signature without streaming raw waveforms.

On-device AI and embedded AI mean the same thing at the device level. TinyML is the subset of edge AI that runs on microcontrollers, often on a battery, in kilobytes of memory.

Edge AI vs cloud AI: what is the difference?

The difference is where inference happens, and that one choice changes latency, connectivity needs, privacy, power and cost.

Factor	Edge AI (on device or gateway)	Cloud AI
Where inference runs	Microcontroller, processor/NPU or local gateway	Remote servers
Response time	Fast and predictable; no network round trip	Depends on the network; adds a round trip and can vary
Works offline	Yes	No; needs a connection for every decision
Data leaving the device	Results or summaries only	Raw or pre-processed data
Bandwidth and radio power	Low: send events, not streams	High for audio, video or high-rate sensor data
Model size and capability	Limited by device memory, compute and power	Very large models are possible
Cost model	Higher bill of materials per unit; low running cost	Low device cost; ongoing compute and data cost per inference
Updating the model	Needs a secure OTA path to every device	Update once on the server
Best for	Real-time control, always-on sensing, private data, poor connectivity	Training, heavy models, fleet-wide analytics, rare complex queries

Where can AI run in a connected product?

There are four practical tiers. Many products use two or three of them together.

Tier	Typical hardware	Memory scale	Models that fit	Example jobs
1. Microcontroller (TinyML)	Arm Cortex-M class MCUs, ESP32, MCUs with a small NPU	Kilobytes to a few megabytes of RAM and flash	Small classifiers and anomaly detectors on sensor data, keyword spotting	Wake word, gesture, vibration anomaly, fall detection
2. Application processor with NPU	Arm Cortex-A SoCs running embedded Linux or Android, often with a neural accelerator	Hundreds of megabytes to gigabytes	Vision models, small speech and language models	Camera analytics, on-device voice commands, quality inspection
3. Gateway or edge server	Industrial PC, IoT gateway, GPU module on site	Gigabytes	Larger vision models, models that combine data from many devices	Line-wide defect detection, site-level predictive maintenance
4. Cloud	Data-centre CPUs and GPUs	Effectively unlimited	Any size, including large language models	Training, retraining, fleet analytics, complex queries

The further down the table you go, the more capability you get and the more you depend on a connection. The further up, the cheaper each decision becomes once the hardware is paid for, and the less data has to travel.

When should inference run on the device?

Run the model at the edge when at least one of these is true. Each is a test you can apply to a specific feature, not to the whole product.

Does the decision have to be fast and predictable?

If the device must react within a control loop, stop a motor, or respond to a user while they wait, a network round trip is a risk. Local inference has no network delay and no dependency on mobile coverage or Wi-Fi quality.

Must it work without a connection?

Devices in basements, fields, vehicles, hospitals or factories lose connectivity. If a feature must keep working during an outage, it has to run locally. Our IoT connectivity guide covers how much bandwidth each radio really gives you.

Is the raw data sensitive?

Audio from a home, images of people, and health or location data are easier to protect if they never leave the device. Sending only results reduces what can leak in transit or in storage, and simplifies the privacy story for customers in the EU, US and India.

Is the data too big or too costly to send?

Continuous audio, video or high-frequency vibration data can cost more in radio power and data charges than the device is worth. On battery-powered products on NB-IoT, LTE-M or LoRaWAN, streaming is often impossible. Local inference turns megabytes into a few bytes of result.

Does the volume make cloud inference expensive?

A feature that runs every second on every device in a large fleet adds up to a large and ongoing cloud bill. Edge inference moves that cost into the device bill of materials once.

When should the model stay in the cloud?

Keep it in the cloud when the model is too large for any affordable device, when the decision is rare and can wait, when it needs data from many devices at once, or when the model changes so often that pushing it to every device would be impractical. Training and retraining almost always stay in the cloud.

What is TinyML, and what can a microcontroller really do?

TinyML is machine learning that runs on microcontrollers: low-cost, low-power chips with kilobytes of RAM, usually without an operating system. It is what makes always-on sensing possible on a coin cell or a small battery.

The tooling is mature enough for production:

LiteRT for Microcontrollers (formerly TensorFlow Lite for Microcontrollers) has a core runtime that fits in about 16 KB on an Arm Cortex-M3 and needs no operating system, no standard C or C++ libraries and no dynamic memory allocation. It supports only a subset of operations and does not support on-device training, so models have to be designed with that in mind.
CMSIS-NN provides optimised neural-network kernels for Arm Cortex-M cores.
MCUs with neural accelerators are now common. Arm's Ethos-U55 microNPU, for example, pairs with Cortex-M55 and Cortex-M85 cores; Arm quotes up to a 480x uplift in ML performance over earlier Cortex-M systems (a vendor figure, not a benchmark).

Good TinyML jobs are narrow: is this sound a wake word, is this vibration normal, is this motion a fall. Open-ended tasks such as free-form speech or chat belong on a bigger processor or in the cloud.

How do you fit a model onto a device?

A model trained in the cloud is usually too large and too slow to run on a device as it is. Four techniques close the gap:

Quantisation. Store weights and activations as 8-bit integers instead of 32-bit floats. Google reports that full integer quantisation makes models about 4x smaller and up to 2–4x faster on CPU, with small accuracy loss in its examples. Integer-only models are also what many NPUs and MCUs need.
Pruning. Remove weights or channels that contribute little, then fine-tune.
Knowledge distillation. Train a small "student" model to copy a large "teacher" model.
Architecture choice. Start from models designed for mobile and embedded use rather than shrinking a server model.

Then check the details that decide whether it ships: every operator in the model is supported by the runtime and accelerator, the model and its working memory fit alongside the application and the OTA image, inference fits the power budget, and accuracy holds on data from the real sensor rather than a public dataset.

What does a hybrid edge-cloud design look like?

Most good designs split the work. Three patterns cover most products:

Filter at the edge, decide in the cloud. The device detects that something interesting happened and sends only that event and a small sample. The cloud runs a larger model on the sample.
Decide at the edge, learn in the cloud. The device acts locally. It uploads selected data, such as low-confidence cases, so the cloud can retrain the model and send an improved version back.
Gateway in the middle. Simple sensors send data to a local gateway that runs heavier models, combines data from many devices and forwards results upstream.

Where do AWS IoT Greengrass and Azure IoT Edge fit?

Both are frameworks for running cloud-managed software, including ML inference, on Linux- or Windows-class edge devices and gateways, not on microcontrollers.

AWS IoT Greengrass deploys components to "core devices". AWS lets you train a model anywhere (for example in SageMaker), store it in S3, and deploy it as a machine-learning component; its sample components use the DLR and TensorFlow Lite runtimes and need at least 500 MB of local storage.
Azure IoT Edge is a feature of Azure IoT Hub. It runs your logic as Docker-compatible container modules on the device, with a runtime that installs, monitors and updates them. Microsoft lists IoT Edge 1.6 LTS as the supported release and states that support for 1.5 LTS ends on 10 November 2026, so teams still on 1.5 should plan an upgrade.

If your product pairs microcontroller-based sensors with a gateway, a common design is TinyML on the sensors and Greengrass or IoT Edge on the gateway. The choice between them usually follows the cloud your backend already runs on.

How do you update AI models on devices in the field?

A model is software, and it will need updating as you collect real-world data. Plan for it from the start:

Ship models through the same secure OTA path as firmware. Sign the model, verify it on the device, install it to a spare slot and roll back if it misbehaves. Our OTA and FOTA updates guide explains how a safe update works step by step.
Version the model separately from the firmware where you can, so a model fix does not need a full firmware release, and record which model every device runs.
Reserve flash and RAM for a larger future model, not just the first one.
Watch for drift. Track confidence scores and field outcomes so you know when the model no longer matches reality.

How do you secure an on-device model?

Putting a model on a device puts it in an attacker's hands. Protect it like the rest of the firmware:

Secure boot and signed updates, so only trusted firmware and models can run.
Encrypted storage or protected memory where the model is valuable intellectual property.
Disable debug ports on production units.
Validate inputs and sanity-check outputs, so a manipulated sensor signal cannot drive a dangerous action without a second check.

Common edge AI mistakes

Choosing the chip before the model. Prototype the model first, measure its size and compute, then pick hardware with headroom.
Testing only on clean datasets. Real sensors have noise, drift and mounting differences. Collect data from the actual device early.
Forgetting power. An inference every second on a battery product must be measured, not estimated.
No update path for models. A model that cannot be updated freezes at launch-day accuracy.
Putting everything at the edge. If a decision is rare and can wait, the cloud may be cheaper and simpler.

Edge AI and cloud AI: frequently asked questions

What is edge AI in simple terms? Edge AI means a device runs its own AI model locally to make decisions from its sensor data, instead of sending that data to the cloud and waiting for an answer.

What is the difference between edge AI and cloud AI? Edge AI runs inference on the device or a nearby gateway, so it is fast, works offline and keeps raw data local. Cloud AI runs inference on remote servers, so it can use much larger models but needs a connection and sends data off the device.

Is edge AI better than cloud AI? Neither is better in general. Edge AI is better for real-time, always-on, private or offline decisions; cloud AI is better for training, very large models and analysis across a whole fleet. Most products combine them.

What is TinyML? TinyML is machine learning on microcontrollers: small models, often 8-bit quantised, running in kilobytes of memory on low-power chips, typically for tasks such as wake-word detection, gesture recognition or vibration anomaly detection.

Can you add edge AI to an existing device? Often, yes. If the current microcontroller has spare memory and compute, a small model can be added with a firmware update. If not, the usual routes are a hardware revision with an AI-capable MCU or processor, or adding a gateway that runs the model.

Do edge AI models need updates after launch? Yes. Real-world data shows where the model is wrong, so plan a secure OTA path for models from the start, with signing, versioning and rollback.

Planning edge AI for your device?

akranIQ builds the firmware, on-device models, gateways and cloud back ends that make edge AI work in shipped products, from TinyML on microcontrollers to embedded Linux with an NPU. If you want to add AI to a device you already make, or decide what should run on the device and what in the cloud, talk to our engineering team or see how our AIoT and edge AI development team in India works. For the wider firmware and connectivity work, see our embedded software and IoT development services.

Edge AI vs Cloud AI: Where Should Your Device Run Its AI?