Packet-Level Network Analytics without Compromises

Network analytics has been a key component of network management for decades. As we look to integrate more intelligence, whether for increased security or to bette …

Network analytics has been a key component of network management for decades. As we look to integrate more intelligence, whether for increased security or to better handle the emergence of new applications like IoT, we need more information from the network and better tools to process the information. Traditionally, network monitoring and analytics systems rely on aggregation (e.g., flow records) or sampling to cope with high packet rates. This has the downside that, in doing so, we lose data granularity and accuracy, and in general limit the possible network analytics we can perform. Recent proposals leveraging software-defined networking or programmable hardware provide more fine-grained, per-packet monitoring but still are based on the fundamental principle of data reduction in the network, before analytics. Even today, modern network analytics system are still incapable of efficiently processing the deluge of information available with fine grained information. In this talk, we present our work to drastically increase software performance for analytics, and to leverage modern programmable switches to generate per-packet information at Terabit line rates. We will present our system which is a complete network monitoring solution that provides insight into every single packet at data center scale traffic rates. Our system consists of a hardware-software co-design leveraging programmable forwarding engines for telemetry and modern parallel programming techniques for analytics. Our system is able to collect and analyze packet records at terabit speeds for 10s of millions of packets per second per application. These applications can easily be parallelized and scale almost linearly with CPU core count. Analytics applications can be written in standard C++ code and can dynamically scale at runtime.