December 26, 2019

223 words 2 mins read

Cost-effective Presto on AWS with Spot nodes

Cost-effective Presto on AWS with Spot nodes

Did you know you can run Presto in AWS at a tenth of the cost with AWS Spot nodes, with just a few architectural enhancements to Presto. Shubham Tagra explores the gaps in Presto architecture, explains how to use Spot nodes, covers enhancements, and showcases the improvements in terms of reliability and TCO achieved through them.
Talk Title Cost-effective Presto on AWS with Spot nodes
Speakers Shubham Tagra (Qubole)
Conference Strata Data Conference
Conf Tag Big Data Expo
Location San Francisco, California
Date March 26-28, 2019
URL Talk Page
Slides Talk Slides
Video

Presto is gaining popularity for interactive and reporting use cases owing to its performance and ease of use. Presto is performant because it executes pipelined in-memory query plans, but its default architecture isn’t amenable to handling Spot node loss. However, many users still use Spot nodes for their cost benefits and tolerate query failures as a necessary evil. With Spot termination notification listeners and its query retry system, Presto improves the user experience. The changes also allow users who cannot tolerate failures to use Spot nodes and enjoy their cost benefits. Shubham Tagra explores the gaps in Presto architecture, explains how to use Spot nodes, covers enhancements like Spot termination notification listeners and the query retry system, and showcases the improvements in terms of reliability and TCO achieved through them.

reliability aws use case performance pipeline
comments powered by Disqus
Data processing at the speed of 100 Gbps using Apache Crail

Data processing at the speed of 100 Gbps using Apache Crail

December 26, 2019

Modern networking and storage technologies like RDMA or NVMe are finding their way into the data center. Patrick Stuedi offers an overview of Apache Crail (incubating), a new project that facilitates running data processing workloads (ML, SQL, etc.) on such hardware. Patrick explains what Crail does and how it benefits workloads based on TensorFlow or Spark.
Serverless for data and AI

Serverless for data and AI

December 21, 2019

What is serverless, and how can it be utilized for data analysis and AI? Avner Braverman outlines the benefits and limitations of serverless with respect to data transformation (ETL), AI inference and training, and real-time streaming. This is a technical talk, so expect demos and code.
Intro + Deep Dive: CoreDNS

Intro + Deep Dive: CoreDNS

October 11, 2019

Best known for its ability to serve as the cluster DNS for Kubernetes, CoreDNS is a plugin-based DNS server that is easy to use and extend, either through the configuration of Corefile, or by adding n …
Building high-performance text classifiers on a limited labeling budget

Building high-performance text classifiers on a limited labeling budget

December 26, 2019

Robert Horton, Mario Inchiosa, and Ali Zaidi demonstrate how to use three cutting-edge machine learning techniquestransfer learning from pretrained language models, active learning to make more effective use of a limited labeling budget, and hyperparameter tuning to maximize model performanceto up your modeling game.
Executive Briefing: What it takes to use machine learning in fast data pipelines

Executive Briefing: What it takes to use machine learning in fast data pipelines

December 25, 2019

Your team is building machine learning capabilities. Dean Wampler demonstrates how to integrate these capabilities in streaming data pipelines so you can leverage the results quickly and update them as needed and covers challenges such as how to build long-running services that are very reliable and scalable and how to combine a spectrum of very different tools, from data science to operations.
Live Aggregators: A scalable, cost-effective, and reliable way of aggregating billions of messages in real time

Live Aggregators: A scalable, cost-effective, and reliable way of aggregating billions of messages in real time

December 23, 2019

Osman Sarood and Chunky Gupta discuss Mists real-time data pipeline, focusing on Live Aggregators (LA)a highly reliable and scalable in-house real-time aggregation system that can autoscale for sudden changes in load. LA is 80% cheaper than competing streaming solutions due to running over AWS Spot Instances and having 70% CPU utilization.