November 8, 2019

138 words 1 min read

Building reliable real-time services with Apache DistributedLog

Building reliable real-time services with Apache DistributedLog

Apache DistributedLog (incubating) is a low-latency, high-throughput replicated log service. Sijie Guo shares how Twitter has used DistributedLog as the real-time data foundation in production for years, supporting services like distributed databases, pub-sub messaging, and real-time stream computing and delivering more than 1.5 trillion (17 PB) events per day.


Talk Title	Building reliable real-time services with Apache DistributedLog
Speakers	Sijie Guo (StreamNative)
Conference	Strata + Hadoop World
Conf Tag	Big Data Expo
Location	San Jose, California
Date	March 14-16, 2017
URL	Talk Page
Slides	Talk Slides
Video

Apache DistributedLog (incubating) is a low-latency, high-throughput replicated log service. Sijie Guo shares how Twitter has used DistributedLog as the real-time data foundation in production for years, supporting services like distributed databases, pub-sub messaging, and real-time stream computing and delivering more than 1.5 trillion (17 PB) events per day. Topics include:

messaging database apache twitter

comments powered by Disqus

Cloudy with a chance of on-prem

Cloudy with a chance of on-prem

November 8, 2019

The cloud is becoming pervasive, but it isnt always full of rainbows. Defining a strategy that works for your company or for your use cases is critical to ensuring success. Jim Scott explores different use cases that may be best run in the cloud versus on-premises, points out opportunities to optimize cost and operational benefits, and explains how to get the data moved between locations.

Compressed linear algebra in Apache SystemML

Compressed linear algebra in Apache SystemML

November 7, 2019

Many iterative machine-learning algorithms can only operate efficiently when a large matrix of training data fits in the main memory. Frederick Reiss and Arvind Surve offer an overview of compressed linear algebra, a technique for compressing training data and performing key operations in the compressed domain that lets you build models over big data with small machines.

Debugging Apache Spark

Debugging Apache Spark

November 7, 2019

Much of Apache Sparks power comes from lazy evaluation along with intelligent pipelining, which can make debugging more challenging than on traditional distributed systems. Holden Karau and Joey Echeverria explore how to debug Apache Spark applications, the different options for logging in Sparks variety of supported languages, and some common errors and how to detect them.

Docker on YARN

November 6, 2019

Docker makes it easy to bundle an application with its dependencies and provide full isolation, and YARN now supports Docker as an execution engine for submitted applications. Daniel Templeton explains how YARN's Docker support works, why you'd want to use it, and when you shouldn't.

From rivulets to rivers: Elastic stream processing in Heron

From rivulets to rivers: Elastic stream processing in Heron

November 6, 2019

Twitter processes billions of events per day the instant the data is generated using Heron, an open source streaming engine tailored for large-scale environments. Bill Graham, Avrilia Floratau, and Ashvin Agrawal explore the techniques Heron uses to elastically scale resources in order to handle highly varying loads without sacrificing real-time performance or user experience.

How Spark can fail or be confusing and what you can do about it

How Spark can fail or be confusing and what you can do about it

November 5, 2019

Just like any six-year-old, Apache Spark does not always do its job and can be hard to understand. Yin Huai looks at the top causes of job failures customers encountered in production and examines ways to mitigate such problems by modifying Spark. He also shares a methodology for improving resilience: a combination of monitoring and debugging techniques for users.