Quantization Techniques for Mobile Models
Learn how to reduce model size by 80% without losing accuracy. Covers post-training quantization and quantization-aware training approaches.
Building a real-time inference pipeline isn't just about deploying a model and calling it done. It's a complete system where data flows from sensors, through processing layers, into your model, and back out to applications that need to act on predictions immediately. We're talking milliseconds matter here.
The challenge? Edge devices aren't cloud servers. They've got limited CPU, memory constraints, and unreliable connectivity. You can't retry failed inferences five seconds later if your application needs results now. This guide walks through what we've learned building these systems for Ottawa startups working with IoT sensors, robotics, and real-time vision processing.
Understanding Your Data Pipeline
Before your model sees any data, you need a reliable way to collect it from edge devices. We typically work with three data patterns: streaming from sensors (temperature, pressure, accelerometer data), batched uploads (images from cameras every 5-10 seconds), and event-driven triggers (motion detection that sends a frame immediately).
The first question we always ask: does your data need to be buffered? Most edge systems can't handle constant inference. If you're running a model every 100 milliseconds on a device with limited power, you'll drain the battery in hours. We usually implement a simple queue with a configurable sampling rate. If you're processing temperature data, sampling every second is fine. If you're detecting objects in video, you might sample every fifth frame. That's not loss—that's smart resource allocation.
Key Implementation Detail
Use a circular buffer for incoming data. Fixed memory allocation, no garbage collection pauses. When it's full, new data overwrites the oldest entries. This prevents memory leaks that'll crash your system after 48 hours of continuous operation.
Educational Context
This guide provides technical information for educational purposes. Real-time inference systems vary significantly based on hardware, software, and use case requirements. Always test thoroughly in your specific environment before deploying to production. Consult with system engineers and test across your actual edge devices to validate performance, latency, and reliability under your exact conditions.
Model Serving on the Edge
Once data's ready, you need to actually run inference. This is where most people stumble. They quantize a model, shave off 80% of the size, deploy it, and then wonder why latency is still 2 seconds on a Jetson Nano. The issue? Model size isn't your only problem. Memory bandwidth, cache misses, and thread overhead all matter just as much.
We typically use TensorFlow Lite or ONNX Runtime. Both are lightweight. Both support quantization. The real difference is in ecosystem maturity—TF Lite's been battle-tested longer, but ONNX Runtime gives you more flexibility if you're working with mixed model types. Don't overthink it. Pick one, optimize your model, measure latency with your actual data, then decide if you need to switch.
Handling Failures Gracefully
Here's the thing nobody mentions until you're three months into production: edge devices fail. Networks drop. Models crash. Your inference pipeline needs to handle all of this without taking down the whole system. We've seen systems where a single inference error crashed the main application loop. That's a design failure, not a data failure.
Build timeout mechanisms. If inference takes longer than 500 milliseconds (or whatever your threshold is), return a cached prediction from the last successful run. Implement fallback models—a lighter, faster model that runs if your primary model fails. Log every error with context, but don't let logging slow down inference. Use asynchronous logging to a buffer that flushes when the system has spare CPU cycles.
1
Implement Input Validation
Check data shape, range, and NaN values before inference. Reject bad data immediately.
2
Set Strict Timeouts
Run inference in a separate thread with a hardcoded timeout. Kill the thread if it exceeds your budget.
3
Cache Results Strategically
Keep predictions from the last 10 successful inferences. Use them as fallback if inference fails.
4
Monitor Memory Usage
Track peak memory during inference. If it exceeds 80% of available RAM, trigger garbage collection.
Measuring What Actually Matters
You'll see benchmarks claiming 50ms latency. Then you deploy and get 200ms. The gap? Real-world conditions. Concurrent processes, memory pressure, thermal throttling—none of this shows up in lab tests. That's why we measure on actual hardware, with actual data, in the actual environment where it'll run.
Track three metrics consistently: p95 latency (95th percentile—that's when your system gets slow), throughput (inferences per second), and memory peak usage. P95 matters more than average because that's when your application might miss a deadline. If your average latency is 30ms but p95 is 500ms, you've got a problem. That's not good performance with occasional spikes—that's a bottleneck you haven't found yet.
Putting It All Together
Real-time inference on edge devices is fundamentally different from cloud ML. You're not optimizing for accuracy—you're optimizing for latency, power, and reliability under constraints. The systems that work don't get there by accident. They get there through careful design: circular buffers for data, lightweight runtime for serving, timeout mechanisms for failures, and obsessive measurement of actual performance.
Start with a simple architecture. Get it working. Measure everything. Then optimize where measurements show you're slow. Don't assume anything about performance until you've profiled it on your actual hardware with your actual data. That's how you build pipelines that don't just work in theory—they work when it matters.
