IoT in the enterprise: A look at Intel (IoT) Inside

Moty Fania shares Intels IT experience implementing an on-premises big data IoT platform for internal use cases. This unique platform was built on top of several open source technologies and enables highly scalable stream analytics with a stack of algorithms such as multisensor change detection, anomaly detection, and more.

Recent years have seen significant evolution of the Internet of Things (IoT). It has become increasingly easy to connect devices to the Internet and send readings to the public cloud in real time. However, it’s quite evident that the adoption of IoT platforms within the enterprise is lagging and less prevalent, due, in part, to the challenges of an on-premises deployment, the lack of related infrastructure, and in some cases, the lack of domain expertise required to deploy analytical solutions that demonstrate high value through real-life use cases. Intel IT has implemented a platform that enables near-real-time stream analytics at scale, using Docker and Core OS for on-premises deployments. Unlike other IoT analytics implementations, which settle for basic statistics or make many assumptions on the collected data, usually relevant only to specific domains, Intel implemented a generic analytics layer that uses machine-learning techniques and advanced statistical tests to provide meaningful insights to enterprise users in different use cases and business domains. Moty Fania shares Intel’s IT experience implementing this on-premises big data IoT platform for internal use cases. Intel’s ultimate goal was to allow users and organizations in the enterprise to gain insights and business value while hiding the complexity and avoiding the tremendous efforts in building, maintaining, and stabilizing such systems. For this purpose, Intel developed a “smart data pipe,” which allows analysis of massive data streams from many devices simultaneously in near real time. Based on several open source technologies including Kafka, Spark, and Hadoop (HBase), this platform is optimized to be portable and easily deployed with Docker and Core OS with the click of a button. To enable stream analytics at scale, Intel implemented a sophisticated stream-processing framework, Pigeon, based on Akka. This model enables us to build a distributed, multithreaded system without the complexities of traditional multithreaded programming. Pigeon implements a cluster capable of processing channels named topologies that process the data according to any arbitrary logic determined by the users. It handles the creation of topologies, balancing them across the cluster, and allows nodes to join or leave dynamically. In the event that one or more topologies fail, they are automatically recovered without any human intervention. Intel also provides an analytical layer on top of this smart data pipe that yields meaningful insights for diverse users and address situations where there is no prior knowledge about the time series data before it starts arriving and there is no way to assume “normal” behavior or distribution. This analytical layer includes four main components: Moty shares the results when the presented analytic capabilities were applied to several datasets—both internally and externally. For example, in one POC, they helped identify a fab tool which caused a yield problem; in another POC, they showed malfunctions of electrical network voltage-sensors. This is a promising direction for enabling the anticipated IOT analytics revolution in the enterprise.