Cloud inference is expensive, slow, and dependent on connectivity. Here's how we deploy computer vision models on edge devices for real-world applications.
Most computer vision demos run in the cloud. Most real-world computer vision deployments run at the edge. The gap between these two environments is where a lot of projects fail.
This post covers what we've learned deploying CV models on embedded hardware — from retail smart fridges to industrial inspection systems.
The business case for edge deployment is usually one or more of:
Latency. A round-trip to a cloud API takes 50–300ms. An edge model runs in 10–50ms. For real-time applications (robotics, industrial safety, retail checkout), this difference is significant.
Cost. Cloud inference at scale is expensive. A camera that runs 24/7 sending frames to a cloud API generates enormous API costs. An edge model runs for free after the hardware cost.
Connectivity. Industrial environments, remote locations, or low-bandwidth settings can't rely on constant cloud connectivity. Edge inference runs offline.
Privacy. Some customers require that no images leave their premises. Edge inference keeps data local.
The main options for edge CV deployment:
| Hardware | TOPS | Power | Typical Use | |---|---|---|---| | Raspberry Pi 5 | 0 (CPU) | 5W | Lightweight detection | | NVIDIA Jetson Orin Nano | 40 | 10W | Full CV workloads | | Coral TPU | 4 | 2W | Ultra-low-power | | Hailo-8 | 26 | 2.5W | Efficient inference | | Intel Neural Compute Stick | 4 | 1W | USB-based inference |
The right choice depends on your model size, required throughput, power constraints, and cost per unit.
For our smart fridge computer vision project, we used Raspberry Pi hardware — sufficient for the inference load and low cost per deployment unit.
A model that runs fine on a cloud GPU often can't run on edge hardware without optimization. The main techniques:
Reduce model weights from 32-bit floats to 8-bit integers (INT8) or even 4-bit. This cuts model size by 4x and inference time by 2-3x with minimal accuracy loss.
# TensorFlow Lite quantization
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.float16]
tflite_model = converter.convert()
Remove model weights that contribute little to output accuracy. Can reduce model size by 50–90% with careful implementation.
Train a small "student" model to mimic a large "teacher" model. The student is much smaller but retains most of the teacher's accuracy.
Start with architectures designed for edge: MobileNetV3, EfficientDet-Lite, YOLOv8n. These are engineered to be fast and small. Don't start with ResNet-50 and optimize down — start small.
A typical edge CV deployment stack we use:
Camera → Frame capture → Preprocessing → Model inference → Post-processing → Action/report
Frame capture: OpenCV or platform-specific camera APIs. Critical: don't capture faster than you can process. Dropped frames are wasted compute.
Preprocessing: Resize, normalize, color conversion. Do this as efficiently as possible — preprocessing is often the bottleneck on underpowered hardware.
Model inference: TensorFlow Lite, ONNX Runtime, or platform-specific runtimes (TensorRT for NVIDIA, CoreML for Apple). Match the runtime to your hardware for maximum speedup.
Post-processing: NMS (non-maximum suppression) for detection, thresholding for classification. Can run on CPU even if inference uses an accelerator.
Reporting: Batch results before sending to cloud. Don't send every frame — send events (detected product, detected anomaly). This dramatically reduces bandwidth.
This is where most teams underinvest. An edge device running in a retail fridge or factory floor is not easily accessible for debugging. You need:
We use MQTT for telemetry from edge devices to a cloud dashboard. It's lightweight and works over unreliable connections.
Edge hardware forces explicit trade-offs. A model with 95% accuracy at 3 FPS may be less useful than one with 89% accuracy at 30 FPS, depending on the application.
Build a test harness that measures both accuracy and inference time on the target hardware — not on your laptop, not on a server. Hardware-specific optimizations (INT8 quantization, hardware-specific ops) often don't apply uniformly across devices.
Test on the actual hardware early. We've had models run at 30 FPS on a Jetson and 2 FPS on a Raspberry Pi for the same task. Don't assume.
Temperature management matters. Embedded hardware throttles when hot. Budget for cooling, especially in enclosed environments.
Camera quality is the floor. A $10 camera with poor low-light performance will limit accuracy no matter how good your model is. The camera is part of the system.
Plan for model updates. Your model will need retraining when the product assortment changes, when detection accuracy degrades, or when you add new classes. Make the update pipeline easy.
The user experience of the edge device matters. If the device needs rebooting regularly or has confusing status indicators, it won't be used correctly in the field.
We've deployed computer vision systems across retail, industrial, and agricultural settings. If you're building an edge CV system, let's talk about the right approach for your hardware and accuracy requirements.