A high-performance system for deep learning inference and visual inspection

Moty Fania explains how Intel implemented an AI inference platform to enable internal visual inspection use cases and shares lessons learned along the way. The platform is based on open source technologies and was designed for real-time streaming and online actuation.

Recent years have seen significant evolvement of deep learning and AI capabilities. AI solutions can augment or replace mundane tasks, increase workforce productivity, and relieve human bottlenecks. Unlike traditional automation, these solutions include cognitive aspects that used to require human decision making. In some cases, deep learning has proven to be even more accurate than humans in identifying patterns and therefore can be effectively used to enable various kinds of automated, real-time decision making. The advanced analytics team at Intel IT recently implemented an internal visual inference platform—a high-performance system for deep learning inference—designed for production environments. This innovative system enables easy deployment of many DL models in production while enabling a closed feedback loop were data flows in and decisions are returned through a fast REST API. The system maximizes throughputs through batching and smart in-memory caching and can be deployed as either a cluster or standalone node. Moty Fania explains how Intel implemented the platform and shares lessons learned along the way. To enable stream analytics at scale, the system was built in a modern microservices architecture using cutting-edge technologies, such as TensorFlow, TensorFlow Serving, Redis, Flask, and more. It is optimized to be easily deployed with Docker and Kubernetes and cuts down time to market for deploying a DL solution. By supporting different kinds of models and various inputs, including images and video streams, this system can enable deployment of smart visual inspection solutions with real-time decision making. Topics include: