Flyte: Cloud Native Machine Learning & Data Processing Platform
Flyte is the backbone for large-scale Machine Learning and Data Processing (ETL) pipelines at Lyft. It is used across business critical applications ranging from ETA, Pricing, Mapping, Autonomous, etc …
| Talk Title | Flyte: Cloud Native Machine Learning & Data Processing Platform |
| Speakers | Haytham AbuelFutuh (Software Engineer, Lyft), Ketan Umare (Senior Staff Software Engineer, Lyft) |
| Conference | KubeCon + CloudNativeCon North America |
| Conf Tag | |
| Location | San Diego, CA, USA |
| Date | Nov 15-21, 2019 |
| URL | Talk Page |
| Slides | Talk Slides |
| Video | |
Flyte is the backbone for large-scale Machine Learning and Data Processing (ETL) pipelines at Lyft. It is used across business critical applications ranging from ETA, Pricing, Mapping, Autonomous, etc. At its core is a Kubernetes native workflow engine that executes 10M+ containers per month as part of thousands of workflows. The talk will focus on,- Architecture of Flyte and its specification language to orchestrate compute and manage data flow across disparate systems like Spark, Flink, Tensorflow, Hive, etc.- Deploying highly scalable and fault tolerant Kubernetes Operators- Learnings from operating Flyte across multiple Kubernetes clusters and using other CNCF technologies like gRPC, Envoy, FluentD, Kustomize and Prometheus.- Use-cases where Flyte can be leveragedThe talk will conclude with a demo of a machine learning pipeline built using the open source version of Flyte.
