December 10, 2019

357 words 2 mins read

Distributed training of deep learning models

Distributed training of deep learning models

Mathew Salvaris, Miguel Gonzalez-Fierro, and Ilia Karmanov offer a comparison of two platforms for running distributed deep learning training in the cloud, using a ResNet network trained on the ImageNet dataset as an example. You'll examine the performance of each as the number of nodes scales and learn some tips and tricks as well as some pitfalls to watch out for.
Talk Title Distributed training of deep learning models
Speakers Mathew Salvaris (Microsoft), Miguel Gonzalez-Fierro (Microsoft), Ilia Karmanov (Microsoft)
Conference Strata Data Conference
Conf Tag Making Data Work
Location London, United Kingdom
Date May 22-24, 2018
URL Talk Page
Slides Talk Slides
Video

In the last year, there have been a number of attempts to train deep CNNs on the ImageNet dataset in the shortest time possible. (The most recent attempt managed to do it in 15 minutes.) But all of these attempts happen on custom clusters, which are out of the reach of most data scientists. One of the key advantages of the cloud is being able to scale out compute resources as required. Mathew Salvaris, Miguel Gonzalez-Fierro, and Ilia Karmanov offer a comparison of two platforms for running distributed deep learning training in the cloud. Both utilize Docker containers, making it possible to run any deep learning framework on them. You’ll examine the performance of each as the number of nodes scales and learn some tips and tricks as well as some pitfalls to watch out for. The first is a service called Batch AI, which uses the Azure Batch infrastructure to easily run deep learning jobs at scale across GPUs. The second is an open source toolkit that allows data scientists to spin up clusters in a turnkey fashion. It utilizes Kubernetes and Grafana for easy job scheduling and monitoring. This solution has been used in daily production for Microsoft internal groups. Mathew, Miguel, and Ilia use these training platforms to train a ResNet network on the ImageNet dataset using each of the following frameworks: CNTK, TensorFlow (Horovod), PyTorch, MxNet, and Chainer. They then compare and contrast the performance. The examples presented can also be used as templates for your own deep learning problems.

gpu cloud cnn mxnet cluster azure dataset tensorflow network container framework math performance grafana pytorch microsoft infrastructure open source deep learning docker monitoring template kubernetes
comments powered by Disqus
Distributed deep learning with containers on heterogeneous GPU clusters

Distributed deep learning with containers on heterogeneous GPU clusters

November 26, 2019

Deep learning model performance relies on underlying data. Dong Meng offers an overview of a converged data platform that serves as a data infrastructure, providing a distributed filesystem, key-value storage and streams, and Kubernetes as orchestration layer to manage containers to train and deploy deep learning models using GPU clusters.
Data science in the cloud

Data science in the cloud

November 27, 2019

In this talk Alex will discuss lessons learned from AWS SageMaker, an integrated framework for handling all stages of analysis. AWS uses open source components such as Jupyter, Docker containers, Python and well established deep learning frameworks such as Apache MxNet and TensorFlow for an easy to learn workflow.
The Path to GPU as a Service in Kubernetes

The Path to GPU as a Service in Kubernetes

November 26, 2019

TLDR: Kubernetes modern production patterns for Deep Learning applications and a deep dive into the Kubernetes GPU and Hardware accelerators subsystem and its challenges (performance, scheduling, mon …
Deep learning with TensorFlow and Spark using GPUs and Docker containers

Deep learning with TensorFlow and Spark using GPUs and Docker containers

December 10, 2019

In the past, you needed a high-end proprietary stack for advanced machine learning, but today, you can use open source machine learning and deep learning algorithms available with distributed computing technologies like Apache Spark and GPUs. Nanda Vijaydev and Thomas Phelan demonstrate how to deploy a TensorFlow and Spark with NVIDIA CUDA stack on Docker containers in a multitenant environment.
Operationalize deep learning models for fraud detection with Azure Machine Learning Workbench

Operationalize deep learning models for fraud detection with Azure Machine Learning Workbench

December 6, 2019

Advancements in computing technologies and ecommerce platforms have amplified the risk of online fraud, which results in billions of dollars of loss for the financial industry. This trend has urged companies to consider AI techniques, including deep learning, for fraud detection. Francesca Lazzeri and Jaya Mathew explain how to operationalize deep learning models with Azure ML to prevent fraud.
Deep learning for domain-specific entity extraction from unstructured text

Deep learning for domain-specific entity extraction from unstructured text

November 27, 2019

Mohamed AbdelHady and Zoran Dzunic demonstrate how to build a domain-specific entity extraction system from unstructured text using deep learning. In the model, domain-specific word embedding vectors are trained on a Spark cluster using millions of PubMed abstracts and then used as features to train a LSTM recurrent neural network for entity extraction.