Lightning Talk: How Kubernetes is Helpful for Accelerating Machine Learning Research and Engineering [I]

In this lightning talk, the presenter shares his experience on helping machine learning research and engineering with kubernetes. k8s is not only a tool for managing microservices but also helpful for …

In this lightning talk, the presenter shares his experience on helping machine learning research and engineering with kubernetes. k8s is not only a tool for managing microservices but also helpful for executing batch jobs like learning phase of deep learning frameworks and stateful services that provides data for the learning tasks. The presenter and his collaborators has been building and managing k8s cluster for TensorFlow learning tasks and HDFS as its learning data source. In addition, thanks to the pluggable scheduler architecture of k8s, their custom scheduler enshorts execution the learning tasks effecitvely and hides usage of network equipments and complex heterogeneous computational devices (e.g. GPUs) from researchers.