Stream analytics in the enterprise: A look at Intels internal IoT implementation

Moty Fania shares Intels IT experience implementing an on-premises IoT platform for internal use cases. The platform was designed as a multitenant platform with built-in analytical capabilities and based on open source big data technologies and containers. Moty highlights the lessons learned from this journey with a thorough review of the platforms architecture.

Recent years have seen significant evolution of the Internet of Things. It has become increasingly easy to connect devices to the Internet and send sensory data to the public cloud. However, the adoption of IoT platforms and stream analytics within the enterprise is lagging and less prevalent, an effect of the lack of skilled developers required to deploy an on-premises platform and the limited demonstration of high value in real-life use cases. Intel IT has addressed these challenges by implementing an internal IoT platform, with the goal of allowing users and organizations to gain insights and business value from real-time analytics. The platform is based on several open source technologies including Akka, Kafka, and Spark Streaming with a full stack of algorithms including multisensor change detection and anomaly detection. To enable stream analytics at scale, Intel implemented a smart data pipe/stream processing framework, Pigeon, that implements a cluster capable of processing topologies that process the data according to any arbitrary logic determined by the users and is optimized to be easily deployed with Docker and CoreOS, which cuts down development by enabling a single developer to deploy a massive real time, elastic processing cluster with a click of a button. And unlike other IoT analytics implementations that settle for basic statistics or make many assumptions on the collected data, Intel implemented a generic analytics layer that uses machine learning and advanced statistical tests to provide meaningful insights to users in different use cases and business domains. Moty Fania explains how Intel identified the set of characteristics and needs common to many IoT scenarios and made them available in one single reusable platform, offers a thorough overview of the platform’s architecture and related technologies (Akka, Kafka, Spark, Hadoop, etc.), demonstrates how Docker and CoreOS made the on-premises deployment easy, and reviews the generic analytics layer that uses machine learning to provide meaningful insights in different use cases and business domains. Moty concludes by discussing how Intel is using this platform to address problems that are not classical IoT use cases but can benefit from real-time analytics to achieve proactivity and operational excellence.