kafka

Building Robust Streaming Data Pipelines with Apache Spark

Building Robust Streaming Data Pipelines with Apache Spark

January 18, 2020

There are challenges to architecting a solution that will allow for developers to stream data into Kafka and be able to manage dirty data which is always an issue in ETL pipelines. I'd like to share l …
Network Telemetry at Yahoo!

Network Telemetry at Yahoo!

January 18, 2020

Providing 1 billion monthly active users with responsive, rich applications requires a large scale network. Locked within processes running on network devices are …
The rise of the streaming platform

The rise of the streaming platform

January 9, 2020

Streaming platforms have emerged as a popular, new trend, but what exactly is a streaming platform? With Apache Kafka at the core, streaming platforms offer an entirely new perspective on managing the flow of data. Neha Narkhede shares examples of Kafka in action and explains why streaming platforms have become the central nervous system for modern digital businesses.
SMACK Stack and Beyond - Building Fast Data Pipelines

SMACK Stack and Beyond - Building Fast Data Pipelines

January 6, 2020

Our world seems to move faster and faster and so are our requirements for data analytics. For many use cases such as fraud detection or reacting on sensor data the response times of traditional batch …
Thriving under a continuous self-inflicted DDoS attack

Thriving under a continuous self-inflicted DDoS attack

January 6, 2020

New Relic customers send monitoring data to New Relic servers every minutea continuous firehose of data. Drawing on his experience at New Relic, Kevin Beck shares best practices for building a streaming service based on Apache Kafka, self-monitoring for reliability and fault tolerance, and building a DevOps culture that anticipates and prevents outages.
Architecting a next-generation data platform

Architecting a next-generation data platform

January 5, 2020

Using Customer 360 and the IoT as examples, Jonathan Seidman, Mark Grover, and Gwen Shapira explain how to architect a modern, real-time big data platform leveraging recent advancements in the open source software world, using components like Kafka, Impala, Kudu, Spark Streaming, and Spark SQL with Hadoop to enable new forms of data processing and analytics.
Building big data applications on Azure

Building big data applications on Azure

January 4, 2020

As big data solutions are rapidly moving to the cloud, it's becoming increasingly important to know how to use Apache Hadoop, Spark, R Server, and other open source technologies in the cloud. Pranav Rastogi walks you through building big data applications on Azure HDInsight and other Azure services.
Lessons from an AWS migration

Lessons from an AWS migration

January 1, 2020

if(we)'s batch event processing pipeline is different from yours, but the process of migrating it from running in a data center to running in AWS is likely pretty similar. Chris Mills explains what was easier than expected, what was harder, and what the company wished it had known before starting the migration.
Mistakes were made, but not by us: Lessons from a year of supporting Apache Kafka

Mistakes were made, but not by us: Lessons from a year of supporting Apache Kafka

December 31, 2019

Dustin Cote shares his experience troubleshooting Apache Kafka in production environments and explains how to avoid pitfalls like message loss or performance degradation in your environment.
State-of-the-art robot predictive maintenance with real-time sensor data

State-of-the-art robot predictive maintenance with real-time sensor data

December 30, 2019

Mateusz Dymczyk and Mathieu Dumoulin showcase a working, practical, predictive maintenance pipeline in action and explain how they built a state-of-the-art anomaly detection system using big data frameworks like Spark, H2O, TensorFlow, and Kafka on the MapR Converged Data Platform.
Stream all the things!

Stream all the things!

December 30, 2019

While stream processing is now popular, streaming architectures must be more reliable and scalable than ever beforemore like microservice architectures in fact. Dean Wampler defines "stream" based on characteristics for such systems, using specific tools like Kafka, Spark, Flink, and Akka as examples, and argues that big data and microservices architectures are converging.
Stream analytics with SQL on Apache Flink

Stream analytics with SQL on Apache Flink

December 30, 2019

Although the most widely used language for data analysis, SQL is only slowly being adopted by open source stream processors. One reason is that SQL's semantics and syntax were not designed with streaming data in mind. Fabian Hueske explores Apache Flink's two relational APIs for streaming analyticsstandard SQL and the LINQ-style Table APIdiscussing their semantics and showcasing their usage.
Scaling a user delivery network for real-time audience targeting

Scaling a user delivery network for real-time audience targeting

December 13, 2019

Adam Shepard peels back the covers on a user delivery networka worldwide distributed data store powering over 80 billion transactions a day at millisecond speed. Join in to learn about eventually consistent data architectures, tiered and hybrid storage layers, and what it takes to manage that much data at scale.
If you Dont Monitor your Infrastructure, you Dont Own it! Regain Control Thanks to Prometheus [I]

If you Dont Monitor your Infrastructure, you Dont Own it! Regain Control Thanks to Prometheus [I]

December 10, 2019

In the French FedEx company we used Prometheus to monitor the infrastructure. It hosts a CQRS Architecture composed with Kafka, Spark, Cassandra, ElasticSearch, and microservices APIs in scala. This …
Architecting a next-generation data platform

Architecting a next-generation data platform

December 5, 2019

Using Entity 360 as an example, Jonathan Seidman, Ted Malaska, and Mark Grover explain how to architect a modern, real-time big data platform leveraging recent advancements in the open source software world, using components like Kafka, Impala, Kudu, Spark Streaming, and Spark SQL with Hadoop to enable new forms of data processing and analytics.
Executive Briefing: Dealing with device data

Executive Briefing: Dealing with device data

December 3, 2019

In 2007, a computer game company decided to jump ahead of competitors by capturing and using data created during online gaming, but it wasn't prepared to deal with the data management and process challenges stemming from distributed devices creating data. Mark Madsen shares a case study that explores the oversights, failures, and lessons the company learned along its journey.
Modern Big Data Pipelines over Kubernetes [I]

Modern Big Data Pipelines over Kubernetes [I]

December 3, 2019

Big data used to be synonymous with Hadoop, but our ecosystem has evolved over time with new database, streaming and machine learning solutions which dont necessarily benefit from the Hadoop deployme …
Kafka Operator: Managing and Operating Kafka Clusters in Kubernetes [A]

Kafka Operator: Managing and Operating Kafka Clusters in Kubernetes [A]

December 2, 2019

In this talk we will demonstrate an approach to management of kafka clusters in kubernetes deployments. We will show how we can provision kafka clusters and configure it using kubernetes concepts and …
Near-real-time ingest with Apache Flume and Apache Kafka at 1 million-events-per-second scale

Near-real-time ingest with Apache Flume and Apache Kafka at 1 million-events-per-second scale

December 1, 2019

Vodafone UKs new SIEM system relies on Apache Flume and Apache Kafka to ingest over 1 million events per second. Tristan Stevens discusses the architecture, deployment, and performance-tuning techniques that enable the system to perform at IoT-scale on modest hardware and at a very low cost.
Rethinking stream processing with Apache Kafka: Applications versus clusters and streams versus databases

Rethinking stream processing with Apache Kafka: Applications versus clusters and streams versus databases

November 30, 2019

Michael Noll explains how Apache Kafka helps you radically simplify your data processing architectures by building normal applications to serve your real-time processing needs rather than building clusters or similar special-purpose infrastructurewhile still benefiting from properties typically associated exclusively with cluster technologies.
Stream analytics with SQL on Apache Flink

Stream analytics with SQL on Apache Flink

November 30, 2019

Although the most widely used language for data analysis, SQL is only slowly being adopted by open source stream processors. One reason is that SQL's semantics and syntax were not designed with streaming data in mind. Fabian Hueske explores Apache Flink's two relational APIs for streaming analyticsstandard SQL and the LINQ-style Table APIdiscussing their semantics and showcasing their usage.
Evolving and Supporting Stateful, Multi-Tenant Decisioning Applications in Production [A]

Evolving and Supporting Stateful, Multi-Tenant Decisioning Applications in Production [A]

November 29, 2019

With our adoption of Kubernetes at Capital One, we have simultaneously reduced our application delivery time-to-market while providing a common platform for streaming pipelines. We leverage Kubernetes …
Global empire: Building for fun and profit

Global empire: Building for fun and profit

November 24, 2019

To establish a global user base, a product needs to support a variety of locales. The challenge with supporting multiple locales is the maintenance and generation of localized strings. Michelle Casbon explains how open source tools like Scala, Apache Spark, Apache Kafka, and Apache PredictionIO (incubating) provide structure for a scalable localization platform with machine learning at its core.
Stream all the things!

Stream all the things!

November 14, 2019

"Stream" is a buzzword for several things that share the idea of timely handling of neverending data. Big data architectures are evolving to be stream oriented. Microservice architectures are inherently message driven. Dean Wampler defines "stream" based on characteristics for such systems, using specific tools as examples, and argues that big data and microservices architectures are converging.
A contextual real-time bidding engine for search engine marketing

A contextual real-time bidding engine for search engine marketing

November 10, 2019

Mahesh Goud shares success stories using Ticketmaster's large-scale contextual bandit platform for SEM, which determines the optimal keyword bids under evolving keyword contexts to meet different business requirements, and explores Ticketmaster's streaming pipeline, consisting of Storm, Kafka, HBase, the ELK Stack, and Spring Boot.
Achieving real-time ingestion and analysis of security events through Kafka and Metron

Achieving real-time ingestion and analysis of security events through Kafka and Metron

November 10, 2019

Kevin Mao explores the value of and challenges associated with collecting raw security event data from disparate corners of enterprise infrastructure and transforming them into high-quality intelligence that can be used to forecast, detect, and mitigate cybersecurity threats.
Apache Kylin 2.0: From classic OLAP to real-time data warehouse

Apache Kylin 2.0: From classic OLAP to real-time data warehouse

November 9, 2019

Apache Kylin, which started as a big data OLAP engine, is reaching its v2.0. Yang Li explains how, armed with snowflake schema support, a full SQL interface, spark cubing, and the ability to consume real-time streaming data, Apache Kylin is closing the gap to becoming a real-time data warehouse.
Architecting a next-generation data platform

Architecting a next-generation data platform

November 9, 2019

Using Entity 360 as an example, Jonathan Seidman, Ted Malaska, Mark Grover, and Gwen Shapira explain how to architect a modern, real-time big data platform leveraging recent advancements in the open source software world, using components like Kafka, Impala, Kudu, Spark Streaming, and Spark SQL with Hadoop to enable new forms of data processing and analytics.