The Lyft data platform: Now and in the future

Lyfts data platform is at the heart of the company's business. Decisions from pricing to ETA to business operations rely on Lyfts data platform. Moreover, it powers the enormous scale and speed at which Lyft operates. Mark Grover and Deepak Tiwari walk you through the choices Lyft made in the development and sustenance of the data platform, along with what lies ahead in the future.

Lyft’s data platform is at the heart of the company’s business. Decisions all the way from pricing to ETA to business operations rely on Lyft’s data platform. Moreover, it powers the enormous scale and speed at which Lyft operates. Mark Grover and Deepak Tiwari cover the technologies Lyft uses for ETL, ad hoc querying, stream ingestion, stream processing, visualization, ML model training, and ML model development. Some of these technologies are open source (Hive, Presto, Spark), and some are homegrown (ML model training and model development engines, for example). Mark and Deepak also discuss other core facets of the data platform, including security, data discovery, and lineage, and explain why Lyft adopted open source tools in some cases and why it decided to build its own on others as well as how those choices have evolved over the years. They conclude with a glimpse of what lies ahead in the future.