Benchmarking deep learning inference

Artificial intelligence has had a tremendous impact on various applications at Baidu, including speech recognition and autonomous driving, although the performance requirements for all of these applications are very different. Sharan Narang outlines the challenges in inference for deep learning models and different workloads and performance requirements for various applications.

Artificial intelligence, particularly deep learning, has revolutionized many different applications over the past few years. Research community and the industry is racing toward advancing the field and realizing real-world impact. To help advance deep learning, Baidu released the open source benchmarking tool DeepBench in 2016, which measures performance on deep learning training operations on different hardware. However, the performance characteristics of inference differ significantly from training. In order to broaden the impact of deep learning, it is important to speed up inference for deep learning algorithms. Improvement in inference times can have a significant impact on user experience in applications using deep learning. Sharan Narang outlines the challenges in inference for deep learning models and different workloads and performance requirements for various applications. Along the way, Sharan discusses the key differences between inference and training and various techniques used to speed up deep learning inference.