Building and Managing a Centralized Kubeflow Platform at Spotify
Machine learning workflows within Spotify have been migrated to Kubernetes by adopting Kubeflow and Kubeflow Pipelines. It helps teams increase model development speed and reduce the time to productio …
| Talk Title | Building and Managing a Centralized Kubeflow Platform at Spotify |
| Speakers | Keshi Dai (Senior ML Infra Engineer, Spotify), Ryan Clough (Senior ML Engineer, Spotify) |
| Conference | KubeCon + CloudNativeCon North America |
| Conf Tag | |
| Location | San Diego, CA, USA |
| Date | Nov 15-21, 2019 |
| URL | Talk Page |
| Slides | Talk Slides |
| Video | |
Machine learning workflows within Spotify have been migrated to Kubernetes by adopting Kubeflow and Kubeflow Pipelines. It helps teams increase model development speed and reduce the time to productionize a machine learning model.In this talk, we will demonstrate some best practices Spotify has learned from managing Kubernetes for backend services and apply them to building a centralized Kubeflow platform. We treat infrastructure as code. We establish customizable and repeatable deployment process. Even with a handful of machine learning/data engineers, we are successfully able to manage multiple Kubernetes clusters and machine learning workloads at scale.We will also show how teams at Spotify use Kubeflow platform as a one-stop shop for their machine learning development, which helps them build better products to improve user listening experience.
