November 23, 2019

142 words 1 min read

Lyft's analytics pipeline: From Redshift to Apache Hive and Presto

Lyft's analytics pipeline: From Redshift to Apache Hive and Presto

Lyfts business has grown over 100x in the past four years. Shenghu Yang explains how Lyfts data pipeline has evolved over the years to serve its ever-growing analytics use cases, migrating from the world's largest AWS Redshift clusters to Apache Hive and Presto for solving scalability and concurrency hard limits.
Talk Title Lyft's analytics pipeline: From Redshift to Apache Hive and Presto
Speakers Shenghu Yang (Lyft)
Conference Strata Data Conference
Conf Tag Big Data Expo
Location San Jose, California
Date March 6-8, 2018
URL Talk Page
Slides Talk Slides
Video

Lyft’s business has grown over 100x in the past four years. Shenghu Yang explains how Lyft’s data pipeline has evolved over the years to serve its ever-growing analytics use cases, migrating from the world’s largest AWS Redshift clusters to Apache Hive and Presto for solving scalability and concurrency hard limits. Topics include:

cluster apache aws analytics use case pipeline yang
comments powered by Disqus
Smart agriculture: Blending IoT sensor data with visual analytics

Smart agriculture: Blending IoT sensor data with visual analytics

November 21, 2019

Mike Prorock offers an overview of mesur.io, a game-changing climate awareness solution that combines smart sensor technology, data transmission, and state-of-the-art visual analytics to transform the agricultural and turf management market. Mesur.io enables growers to monitor areas of concern, providing immediate benefits to crop yield, supply costs, farm labor overhead, and water consumption.
Speed up mission-critical analytics in the cloud (sponsored by Kyligence)

Speed up mission-critical analytics in the cloud (sponsored by Kyligence)

November 20, 2019

As organizations look to scale their analytics capability, the need to grow beyond a traditional data warehouse becomes critical, and cloud-based solutions allow more flexibility while being more cost efficient. Billy Liu offers an overview of Kyligence Cloud, a managed Apache Kylin online service designed to speed up mission-critical analytics at web scale for big data.
Architecting an advanced analytics platform for machine learning

Architecting an advanced analytics platform for machine learning

November 18, 2019

Georgios Gkekas shares ING's advanced analytics journey to promote modern machine and deep learning techniques internally through a central, best-of-breed technical platform tailored for data science activities. The platform offers only the necessary automated tools to replace the tedious, repetitive, and error-prone steps in a typical data science pipeline.
Building stream processing as a service at Netflix

Building stream processing as a service at Netflix

November 17, 2019

Steven Wu explains how Netflixs SPaaS platform empowers users to focus on extracting insights from data streams and build stream processing applications and shares lessons learned building and operating the largest SPaaS use case: Netflixs Keystone data pipeline, a self-serve platform for creating near-real-time event pipelines that processes three trillion events and 12 PB of data every day.
Machine learning versus machine learning in production

Machine learning versus machine learning in production

November 23, 2019

Acme Corporation is a global leader in commerce marketing. Manu Mukerji walks you through Acme Corporation's machine learning example for universal catalogs, explaining how the training and test sets are generated and annotated; how the model is pushed to production, automatically evaluated, and used; production issues that arise when applying ML at scale in production; lessons learned; and more.
Whats in the Box? Resource Management in Kubernetes

Whats in the Box? Resource Management in Kubernetes

November 23, 2019

As the cloud native ecosystem evolves, it is determined to align platforms and technologies into a well-orchestrated infrastructure for optimum utilisation of compute, network and storage. This infras …