Encoding 250,000 Songs a Day with batch/v1 Jobs
Tasked with rebuilding the way we deliver music to DJ's, the Beatport Infrastructure team set out to use Kubernetes to construct scalable compute for executing batch and on-demand encoding workloads i …
| Talk Title | Encoding 250,000 Songs a Day with batch/v1 Jobs |
| Speakers | Leigh Capili (DX Engineer, Weaveworks), John Slivka (Infrastructure Engineer, Beatport) |
| Conference | KubeCon + CloudNativeCon North America |
| Conf Tag | |
| Location | Seattle, WA, USA |
| Date | Dec 9-14, 2018 |
| URL | Talk Page |
| Slides | Talk Slides |
| Video | |
Tasked with rebuilding the way we deliver music to DJ’s, the Beatport Infrastructure team set out to use Kubernetes to construct scalable compute for executing batch and on-demand encoding workloads in order to level-up our customer’s capabilities for playing and mixing dance music. What would follow is a 5-month journey of building clusters, thrashing with software dependencies, and trudging through erratic performance and scalability issues with the kubernetes API. How did we decide to use kubernetes? Was it easy to prototype? Is etcd capable of sustainably servicing 10,000 Jobs an hour? How many Pods can the kubernetes API store? How do you monitor and manage Job failures? We’ll walk you through our lessons learned and talk about our most exciting moments and deflating realizations. Join us we re-tell the story of delivering a correct system to production :)
