December 11, 2019

170 words 1 min read

Batch and real-time processing in LINE's log analysis platform

Batch and real-time processing in LINE's log analysis platform

LINEone of the most popular messaging applications in Asiaoffers many services, such as its news application. These services sometimes depend on real-time processing. Wataru Yukawa offers an overview of LINE's web tracking system, which consists of the JavaScript SDK, NGINX Fluentd, Kafka, Elasticsearch, and Hadoop, and explains how it helps with batch and real-time processing.
Talk Title Batch and real-time processing in LINE's log analysis platform
Speakers Wataru Yukawa (LINE)
Conference Strata Data Conference
Conf Tag Making Data Work
Location London, United Kingdom
Date May 22-24, 2018
URL Talk Page
Slides Talk Slides
Video

LINE—one of the most popular messaging applications in Asia—offers many services, such as its news application. These services sometimes depend on real-time processing. Wataru Yukawa offers an overview of LINE’s two-layer log analysis platform, consisting of a batch layer, which uses Hive, Presto, and Hadoop, and a web tracking system, which uses the JavaScript SDK, NGINX Fluentd, Kafka, Elasticsearch, and Hadoop. Wataru focuses on the latter to explain how it helps with batch and real-time processing.

kafka messaging hadoop javascript tracking elasticsearch
comments powered by Disqus
Big data insights equal big money: Stories from the trenches at GoDaddy

Big data insights equal big money: Stories from the trenches at GoDaddy

November 29, 2019

GoDaddy ingests and analyzes over 100,000 data points per second. Felix Gorodishter discusses the company's big data journey from ingest to automation, how it is evolving its systems to scale to over 10 TB of new data per day, and how it uses tools like anomaly detection to produce valuable insights, such as the worth of a reminder email.
Scaling the AI hierarchy of needs with TensorFlow, Spark, and Hops

Scaling the AI hierarchy of needs with TensorFlow, Spark, and Hops

December 5, 2019

Distributed deep learning can increase the productivity of AI practitioners and reduce time to market for training models. Hadoop can fulfill a crucial role as a unified feature store and resource management platform for distributed deep learning. Jim Dowling offers an introduction to writing distributed DL applications, covering TensorFlow and Apache Spark frameworks that make distribution easy.
Moving the needle of the pin: Streaming hundreds of terabytes of pins from MySQL to S3/Hadoop continuously

Moving the needle of the pin: Streaming hundreds of terabytes of pins from MySQL to S3/Hadoop continuously

November 22, 2019

With the rise of large-scale real-time computation, there is a growing need to link legacy MySQL systems with real-time platforms. Henry Cai and Yi Yin offer an overview of WaterMill, Pinterest's continuous DB ingestion system for streaming SQL data into near-real-time computation pipelines to support dynamic personalized recommendations and search indices.
Playing well together: Big data beyond the JVM with Spark and friends

Playing well together: Big data beyond the JVM with Spark and friends

November 22, 2019

Holden Karau and Rachel Warren explore the state of the current big data ecosystem and explain how to best work with it in non-JVM languages. While much of the focus will be on Python + Spark, the talk will also include interesting anecdotes about how these lessons apply to other systems (including Kafka).
Stream processing with Kafka

Stream processing with Kafka

November 20, 2019

Tim Berglund leads a basic architectural introduction to Kafka and walks you through using Kafka Streams and KSQL to process streaming data.
Apache Kafka + Apache Mesos = Highly scalable streaming microservices

Apache Kafka + Apache Mesos = Highly scalable streaming microservices

November 18, 2019

Kai Whner shares a highly scalable, mission-critical infrastructure using Apache Kafka and Apache Mesos: Kafka brokers are used as the distributed messaging backbone; Kafkas Streams API embeds stream processing into any external application without the need for a dedicated streaming cluster; and Mesos is used as a scalable infrastructure to leverage the benefits of a cloud-native platform.